First, I'd like to thank you for running tests for so many
configurations. It is most helpful.
On Sat, Jan 3, 2015 at 7:36 PM, Tom Kent
I've been pondering this recently, but a forced upgrade of my azure VMs last night is pressing the issue...and I wanted to solicit feedback before I go and set things back up...
[snip]
I can try to set things up like this again, but there are two reasons I was thinking about changing: 1) It seems that a lot of the gcc/clang versions have the same pass/fail results as the other versions of that compiler, so we're not adding a ton (though we may be adding the crucial points where a compiler change breaks something...is this something that devs have seen in practice?)
This is true that developer errors tend to influence multiple, if not all compilers. However, this kind of errors is also the easiest to discover and fix. Often the developer is able to reproduce such failure locally. The other kind of problems is compiler bugs. These are naturally specific to the compiler versions, sometimes up to the patch version. The developer often does not have all compiler versions installed, so this kind of problems are hard to discover and fix locally. For this reason having multiple compiler versions in the matrix is invaluable help to developers, and I would very much like to have it. From personal experience, I do stumble upon compiler-specific problems from time to time. Another point is testing compiler-specific code (i.e. the code that uses compiler-specific features available since some version). Our current infrastructure doesn't really suit for that kind of testing, but still it is possible to see that the code compiles and runs. In some cases it is possible to verify that it doesn't generate warnings. If you want to reduce the number of configs to test, you can probably have a look which compiler versions are used in different Linux distros (e.g. CentOS, Fedora, OpenSUSE, Ubuntu, Debian) of the latest LTS and normal releases. For instance, I know gcc 4.4 was used in Debian Squeeze, but in Wheezy (the latest Debian release at the moment) it's 4.7, so if no one else uses 4.4 we can probably drop it. Make a list of what you'd like to drop and post it here so that people can see which versions specifically are candidates for removal.
2) The new version of my runners (Ubuntu 14.04) doesn't easily support the old gcc versions, it would take a lot of effort to get some of the old ones running. Clang is even worse...it is much more difficult to get even two versions running side by side now.
Is it possible to install different versions in chroot environment?
Instead of running lots of different versions, I was thinking about running various compatibility options of the two main compilers...I've seen some other test runners with things like libc++, c++11, c++14, etc. My real question, would developers be better served by these options than different versions? Would they prefer both with longer revisit times between each test type? Other thoughts?
Having testbeds for newer language versions would be good, but if I had to choose between more language versions and more compilers I would choose the latter. It would be ok for me if there were a limited set of fast turnaround testbeds (e.g. latest widespread gcc in C++03 and C++11 modes) and the more exotic rest were less frequent, as long as "less frequent" is reasonable. I'd say, any testbed should cycle at least once a week or so. With longer refresh periods it becomes more difficult to fix bugs before the release.