[testing] More failed test results with no meaningful content
For example: http://www.boost.org/development/tests/develop/developer/output/marshall-mac... leads to a successfully compiled file if you follow the links. Anyone any idea what's going on here, and is there any way to make this reporting more reliable? Frankly the results will get widely ignored if no one can see what's actually going on. Thanks, John.
On Feb 6, 2014, at 12:22 PM, John Maddock wrote:
For example: http://www.boost.org/development/tests/develop/developer/output/marshall-mac... leads to a successfully compiled file if you follow the links.
Anyone any idea what's going on here, and is there any way to make this reporting more reliable? Frankly the results will get widely ignored if no one can see what's actually going on.
We made a recent change to limit how much output is captured from test targets to 4k per target. We can bump this up though it may adversely impact some testers who have more limited bandwidth. How about 16k per target and let us know if that's still insufficient? -- Noel
On Thursday 06 February 2014 19:33:11 Belcourt, Kenneth wrote:
On Feb 6, 2014, at 12:22 PM, John Maddock wrote:
For example: http://www.boost.org/development/tests/develop/developer/output/marshall-> > mac-boost-bin-v2-libs-math-test-test_igamma_inva_float-test-clang-darwin-t ot-debug-link-static.html leads to a successfully compiled file if you follow the links.
Anyone any idea what's going on here, and is there any way to make this reporting more reliable? Frankly the results will get widely ignored if no one can see what's actually going on. We made a recent change to limit how much output is captured from test targets to 4k per target. We can bump this up though it may adversely impact some testers who have more limited bandwidth. How about 16k per target and let us know if that's still insufficient?
I have a similar problem with the output that only shows the library target: http://www.boost.org/development/tests/master/developer/marshall-mac-log-cla... Is it still considered "the test target"? I don't see a note that says that the output is truncated due to size limits. Is it supposed to be still appended? 4k sounds extremely small, and 16k too, BTW. It was 64k before, IIRC, and I did see the note about truncated output from time to time. Some compiler errors may easily produce 100k+ of output and I wouldn't want to lose any of it since there may be useful information in it. Is bandwidth really the problem? Can't we just compress the output in scripts before uploading? The text should be quite well compressible.
Is bandwidth really the problem?
No.
Can't we just compress the output in scripts before uploading?
We already do.
The text should be quite well compressible.
The problem is that the output is so large for some compilers that it overflows the compression programs capabilities to generate an archive that can contain the data.. We are talking of compressing multi-gig results because of multi-gigs of compiler outputs. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On Thursday 06 February 2014 13:56:19 Rene Rivera wrote:
Is bandwidth really the problem?
No.
Can't we just compress the output in scripts before uploading?
We already do.
The text should be quite well compressible.
The problem is that the output is so large for some compilers that it overflows the compression programs capabilities to generate an archive that can contain the data.. We are talking of compressing multi-gig results because of multi-gigs of compiler outputs.
I see. Out of curiosity, what compression programs are those? Maybe it's possible to build and compress the results in parts, e.g. a single archive per library per compiler?
On Thu, Feb 6, 2014 at 2:07 PM, Andrey Semashev
Is bandwidth really the problem?
No.
Can't we just compress the output in scripts before uploading?
We already do.
The text should be quite well compressible.
The problem is that the output is so large for some compilers that it overflows the compression programs capabilities to generate an archive
On Thursday 06 February 2014 13:56:19 Rene Rivera wrote: that
can contain the data.. We are talking of compressing multi-gig results because of multi-gigs of compiler outputs.
I see. Out of curiosity, what compression programs are those?
7za or zip in that preference order. Maybe it's possible to build and compress the results in parts, e.g. a
single archive per library per compiler?
IIRC, the case that prompted the limits some days ago was for *one* compiler generating that much output. And, no, we can't currently limit it per library. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
On 06/02/2014 19:33, Belcourt, Kenneth wrote:
On Feb 6, 2014, at 12:22 PM, John Maddock wrote:
For example: http://www.boost.org/development/tests/develop/developer/output/marshall-mac... leads to a successfully compiled file if you follow the links.
Anyone any idea what's going on here, and is there any way to make this reporting more reliable? Frankly the results will get widely ignored if no one can see what's actually going on.
We made a recent change to limit how much output is captured from test targets to 4k per target. We can bump this up though it may adversely impact some testers who
have more limited bandwidth.
How about 16k per target and let us know if that's still insufficient?
I don't think that is the issue - it's not that the output is truncated - it's that there is *no output at all*. Or rather that the link for the "failure" leads to a file which did actually compile correctly. John.
We made a recent change to limit how much output is captured from test targets to 4k per target. We can bump this up though it may adversely impact some testers who have more limited bandwidth. How about 16k per target and let us know if that's still insufficient?
I don't think that is the issue - it's not that the output is truncated - it's that there is *no output at all*. Or rather that the link for the "failure" leads to a file which did actually compile correctly.
Which is not to say the report truncation isn't an issue, for example: http://www.boost.org/development/tests/develop/developer/output/BP%20x86_64%... displays *one* warning and that's it. What on earth am I supposed to do with that? Thanks, John.
On Feb 7, 2014, at 3:57 AM, John Maddock
We made a recent change to limit how much output is captured from test targets to 4k per target. We can bump this up though it may adversely impact some testers who have more limited bandwidth. How about 16k per target and let us know if that's still insufficient?
I don't think that is the issue - it's not that the output is truncated - it's that there is *no output at all*. Or rather that the link for the "failure" leads to a file which did actually compile correctly.
Which is not to say the report truncation isn't an issue, for example: http://www.boost.org/development/tests/develop/developer/output/BP%20x86_64%... displays *one* warning and that's it. What on earth am I supposed to do with that?
If you have a question about a test that’s running on one of my testers, please drop me a line, and I can run that test and send you the entire output. — Marshall P.S. Just not for the next 8 days.
I don't think that is the issue - it's not that the output is truncated - it's that there is *no output at all*. Or rather that the link for the "failure" leads to a file which did actually compile correctly.
Which is not to say the report truncation isn't an issue, for example: http://www.boost.org/development/tests/develop/developer/output/BP%20x86_64%... displays *one* warning and that's it. What on earth am I supposed to do with that?
If you have a question about a test that’s running on one of my testers, please drop me a line, and I can run that test and send you the entire output.
For sure, but the whole idea of the reporting system is that we shouldn't have to bother the test runners with questions like that. Cheers, John.
On Fri, Feb 7, 2014 at 5:10 AM, John Maddock
I don't think that is the issue - it's not that the output is truncated - it's that there is *no output at all*. Or rather that the link for the "failure" leads to a file which did actually compile correctly.
Is that a symptom of the execution timing out? --Beman
On Feb 7, 2014, at 6:33 AM, Beman Dawes wrote:
On Fri, Feb 7, 2014 at 5:10 AM, John Maddock
wrote: I don't think that is the issue - it's not that the output is truncated - it's that there is *no output at all*. Or rather that the link for the "failure" leads to a file which did actually compile correctly.
Is that a symptom of the execution timing out?
I've noticed this before when a tester runs multiple toolsets, like Marshall's mac, the test report can be misleading or just plain wrong for some libraries. Have you seen the no output problem with single toolset runs?
I don't think that is the issue - it's not that the output is truncated - it's that there is *no output at all*. Or rather that the link for the "failure" leads to a file which did actually compile correctly.
Is that a symptom of the execution timing out?
Well it says "Compile [2014-02-07 13:37:19 UTC]: succeed" so I took it at it's word ;-) If there is a timeout then it should say so, though there shouldn't be - none of those files take more than a couple of seconds to compile here even on my rather underpowered laptop - if that is the case I would suspect that the tests are run on a virtual machine with little memory available - once you start hitting the swapfile on a virtual machine these builds can take more or less "forever", especially if they're run with a high concurrency value. John.
AMDG On 02/07/2014 02:10 AM, John Maddock wrote:
I don't think that is the issue - it's not that the output is truncated - it's that there is *no output at all*. Or rather that the link for the "failure" leads to a file which did actually compile correctly.
The basic problem is that process_jam_log assumes one test case or one library = one source file. Support for incremental builds makes this a silent error. We want to ignore the old output when the target is updated. Thus, when process_jam_log handles a compile.c++ target, it discards any previous output for the same lib target. Unfortunately, this includes output produced when compiling another source file. The result is that the last source file processed is the only one that appears in the report. I looked into fixing this a while back, but it's non-trivial. In Christ, Steven Watanabe
participants (8)
-
Andrey Semashev
-
Belcourt, Kenneth
-
Beman Dawes
-
John Maddock
-
John Maddock
-
Marshall Clow
-
Rene Rivera
-
Steven Watanabe