2013/6/3 Niall Douglas
Dear Boost,
I spent the weekend experimenting with getting CI set up for the async file i/o library which will hopefully become Boost.AFIO through Paul's GSoC. I tried a number of free-for-open-source CI providers which integrate tightly with GitHub, and eventually settled with Travis CI. You can see the last commit CI results at https://travis-ci.org/ned14/TripleGit.
As much as Travis CI has a primitive and clunky UI, unlike the other CIs it's based on snapshotting VMs with root privs which gives you unparalleled flexibility for testing config, so I was able to configure this to happen with each and every git push to github:
1. Test that code builds with GCC 4.6 and clang 3.2.
2. Test that unit and functional tests execute without error when built with GCC 4.6 and clang 3.2.
3. Run unit and functional tests with valgrind --tool=memcheck and test that no memory errors nor memory leaks occur with binaries built with GCC 4.6 and clang 3.2.
4. Install GCC 4.8 and compile everything using -fsanitize=thread (i.e. instrument using the port of clang's ThreadSanitizer v2 to GCC), then execute a full set of unit and functional tests looking for race conditions and if any, fail. You'll note I've marked that test as allowed to fail, because Boost v1.53 has a number of known race conditions when used as C++11 which are hopefully fixed in v1.54 :)
Personally speaking, I think this depth and degree of CI is *invaluable* to modern software development practice.
I could not agree more. One thing you may want to add is coverage analysis with https://coveralls.io/: If you build with gcc, pass "-fprofile-arcs -ftest-coverage" both to the compiler and the linker. After running the tests, you can analyze the coverage with gcov and send it to coveralls. To collect and upload, I use the following script: https://github.com/purpleKarrot/Karrot/blob/develop/test/coveralls.in (Note that gcov 4.6 does not support all flags that I use in the script, I use it wiht 4.8 only). cheers, Daniel