On 29 Jul 2014 at 17:02, Louis Dionne wrote:
Also, regarding formal review, I personally would feel uncomfortable accepting a library that only works with a single version of clang. I would feel much happier if you got trunk GCC working, even if that means workarounds.
That would mean a lot of workarounds.
Not necessarily. You could wait until GCC trunk catches up instead, helping it along by adding however many bug reports as is necessary. Timing well a formal review is often as important as the quality of your library. I always feel heebie jeebies about code which works only on one compiler. For me to vote yes for a Boost library to enter Boost I need to feel it is well tested and reliable and has had all its kinks knocked out. I struggle to see me feeling this with a library which can't be tested widely. Instead of heading straight into the community review queue, perhaps a few rounds of intermediate informal reviews like this one? I'd particularly like to see Eric and Joel's opinion of your library so far too.
Basically, I got a benchmark for almost every operation of almost every sequence that's supported by Hana (including those adapted from external libraries), but I'm not sure yet of how to group them in the documentation (per operation? per sequence? per type class?).
The problem is made worse by two things:
- It only makes sense to benchmark components that are isomorphic. For example, what does it mean to benchmark a std::tuple against a mpl::vector? Not much, because the price you pay for std::tuple gives you the ability to hold values, whereas mpl::vector can only hold types. We don't want to compare apples with oranges, and the grouping of benchmarks should be influenced by that.
I guess it's like when choosing between list, map and vector from the STL. All are "equivalent", but you need to know how each might scale out for what you need. Most users won't just known automatically. My problem always with MPL98 was I had no idea what was fast or slow for my use cases.
- How do we handle different compilers? Right now, all benchmarks are produced only on Clang, which is OK because it's the only compiler that can compile the library. When there is more than one compiler, how do we generate the benchmarks for all of them, and how do we integrate the benchmarks in the documentation?
You're right this is more of a name and shame thing between AST compilers. MSVC is very different though, and you may eventually need a separate line labelled "MSVC" just for it.
I'd also like to see unit testing that verified that the current compiler being tested has a time and space benchmark curve matching what is expected. It is too easy for code to slip in or the compilers themselves to gain a bug which creates pathological metaprogramming performance. Better to have Travis CI trap that for you than head scratching and surprises later.
I thought about doing this, but I did not because I thought it was a HUGE undertaking to automate it.
No, it's easier than you think. Have a look at https://ci.nedprod.com/ whose default dashboard shows a graph labelled "RUDP performance". This tracks performance of a build over time to ensure performance doesn't regress. All you need is for your performance test tool to output some CSV, a Jenkins Plot plugin does the rest.
I think we're better off integrating the benchmarks in the documentation and then when something is really weird, we just have to look at the generated benchmarks and see what's wrong. If someone can suggest a way to do it automatically that won't take me weeks to set up, I'm interested.
Mastering Jenkins takes months, but once mastered configuring all sorts of test scenarios becomes trivial. I'd actively merge your Jenkins/Travis output into your docs too, it is nowadays an online world.
Also, I wouldn't want to slip in the trap of testing the compiler; testing Hana is a large enough task as it is (341 tests + 165 examples as we speak).
I think testing modern C++ is always testing the compiler. It's why I develop first on MSVC, I stand a good chance of porting that to GCC or clang, less so the other way round.
Finally, there are ways and means for doxygen docs to automatically convert into BoostBook docs. You'll need to investigate those before starting a formal review. Tip: look into how Geometry/AFIO does the doxygen conversion, it's brittle but it is easier than the others.
Is it mandatory for a Boost library to have BoostBook documentation? I'd like to stay as mainstream as possible in the tools I use and reduce the number of steps in the build/documentation process for the sake of simplicity. Is there a gain in generating the documentation in BoostBook?
As you'll find when formal review comes, to pass isn't about how good your library is, it's about eliminating as many rational objections others can think of. BoostBook format documentation is a very easy way of crossing permanently off a raft of potential objections. Other categories of objection will be far harder to address, trust me. I suspect the hardest will probably be the "so what?" objection in all the guises it manifests, as it always is for any library which is particularly novel. All that said, I have little love for the Boost documentation format, I think it outdated, brittle, and inflexible, and it does a lousy job of automating reference documentation. But it could be a lot worse, and I have been forced in the past to use a lot worse. Besides, you'll invest five days or so of wishing pain on those responsible for the tools, and once it's working you'll never need touch it again. I did find it took some months to find and fix all the corner cases in the doc output though, and even now PDF generation from AFIO's docs are a joke due to the long template strings. Anyway, it's up to you. BTW, I've noticed that when peer review managers volunteer to manage they tend to favour ones with BoostBook docs. I think they also think it's another problem they don't have to think about during managing. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/