On 1/23/21 10:07 PM, Robert Ramey via Boost wrote:
On 1/23/21 7:38 AM, Andrey Semashev via Boost wrote:
On 1/23/21 3:47 PM, Tom Kent via Boost wrote:
I've got a couple raspberry pi 4's that are running tests (slowly, takes 20+hrs to run the test suite...any earlier models just didn't have enough ram). Look at the teeks99-05* (armv71/armhf) and teeks99-06* (aarch64).
I appreciate your and all other testers efforts in running the test suite, but I must confess that I pay almost no attention to the official test matrix these days because:
The boost test matrix is the most complete and reliable display of the current state of the the boost packages.
Sure, if you want to see the status of the whole Boost. I, as a library maintainer, am more interested in my specific library status, after having pushed a commit. This information is not readily provided by the test matrix.
I do run the more moder CI stuff, but it's often failing for some issue or another totally unrelated to my library. It's the gold standard as far as I'm concerned.
In my experience, the official test matrix is not more reliable than CI, when it comes to random failures. More than once I've seen failures caused by configuration errors on tester's machine (e.g. compiler executable not found). There were also weird failures for no apparent reason, which turned out to be the result of updates on the tester's machine. CI images are more stable, and you also can install necessary dependencies for testing. There are some quirks, and the installation can fail from time to time, but in general I would say the CI errors are more actionable.
- Problematic debugging. Often the report shows a failure, but the error log is not accessible. This seems to be a long standing problem. This makes the whole testing process pointless, as I cannot do anything about the failures.
I have difficulties sifting through the test output on all platforms.
The problem is not too much output (that wouldn't be a problem at all). The problem is that you often get no output at all.
(I've been roundly ridiculed for this complaint. But it means nothing to me - I wear their ridicule as a badge of honor.) I have my own solution which I run on my own machine - library_status which presents a table which is actually more useful to me than the official boost one not to mention the Appveyor one. Now If I could get library_status to run as part of the CI solutions ...
A short status table might be nice, but that is not my complaint. I can do without such a table just fine. I can't do without the build and test output in case of failure.
I wish the current testing infrastructure was replaced with something more modern, CI-style, as I don't believe the above issues will be fixed any time soon.
I've made a worthy proposal for that (to be used in addition to the current boost test matrix). Again, got a lot of ridicule on that one too.
I haven't seen this, sorry.