Re: [boost] [build] Test marked as compile-fail is now reported as a failure.
Hi Vladimir
2015-01-12 7:26 GMT+01:00 Vladimir Prus
Hi Adam,
On 01/11/2015 10:38 PM, Adam Wulkiewicz wrote:
Hi,
Since yesterday one test marked as compile-fail is now reported as a failure on the regression tests page.
It's the test: rtree_values_invalid on the page: http://www.boost.org/development/tests/develop/ developer/geometry-index.html
Surprisingly other tests aren't. see e.g.: geographic_custom_point_with_wrong_units on the page: http://www.boost.org/development/tests/develop/ developer/geometry.html
Why is it surprising that test A fails while test B does not? Are these exactly identical tests?
They're both expected to fail to compile.
They seem to be defined the same way in the Jamfile:
https://github.com/boostorg/geometry/blob/develop/index/ test/rtree/Jamfile.v2 https://github.com/boostorg/geometry/blob/develop/test/ point_concept/Jamfile.v2
Could it be that the problem with the actual tested functionality.
AFAIU no because the test is reported as a fail only if it's expected to compile/link/run but it fails to compile/link/run. If the situation is the opposite (expected to fail but doesn't fail) the test is reported as a pass (with dark green background). Otherwise the test should be reported as a "regular" pass.
AFAIS the oldest run when it failed was of the runner:
teeks99-03c-Ubuntu12.04-64 on Sat, 10 Jan 2015 02:05:44. On Fri, 09 Jan 2015 18:36:55 (teeks99-08b-win2012R2-64on64) everything was ok.
Furthermore when run locally it's also reported as a failure. Some time ago of course "(failed-as-expected)" message was printed instead. Do you have any idea what may be wrong?
Looking at
http://www.boost.org/development/tests/develop/ developer/output/BP%20x86_64%20C++11-boost-bin-v2-libs- geometry-index-test-rtree- rtree_values_invalid-test- clang-linux-3-5~c14_libc++-debug-debug-symbols-off.html
It does look indeed that complication is successfull, while it should fail for the test to pass.
If by successful you mean "the compiler output is present" then indeed it isn't. However it's not always the case, see: http://www.boost.org/development/tests/develop/developer/output/BP%20x86_64%... http://www.boost.org/development/tests/develop/developer/output/trippels_gcc... Though I can't see any error message there, only warnings. But this is not the first time I see something like this on the regression tests page. However the message about trimming of the output is displayed only for the first of the tests mentioned above.
Locally for me, with gcc, the compilation fails, and the test is deemed passing, so I can't see any Boost.Build malfunction here.
Yes, the cause may be located elsewhere, in Geometry, Regression, Test, etc. For a clean run I'm now unable to build the tests due to some undefined references but the problem is most likely located elsewhere.
Am I missing something?
Or am I? Regards, Adam
Hi Adam, On 01/12/2015 06:58 PM, Adam Wulkiewicz wrote:
see e.g.: geographic_custom_point_with_wrong_units on the page: http://www.boost.org/development/tests/develop/ developer/geometry.html
Why is it surprising that test A fails while test B does not? Are these exactly identical tests?
They're both expected to fail to compile.
They seem to be defined the same way in the Jamfile:
https://github.com/boostorg/geometry/blob/develop/index/ test/rtree/Jamfile.v2 https://github.com/boostorg/geometry/blob/develop/test/ point_concept/Jamfile.v2
Could it be that the problem with the actual tested functionality.
AFAIU no because the test is reported as a fail only if it's expected to compile/link/run but it fails to compile/link/run. If the situation is the opposite (expected to fail but doesn't fail) the test is reported as a pass (with dark green background). Otherwise the test should be reported as a "regular" pass.
I am fairly sure that dark green vs green depends only on status/explicit-failures-markup.xml, and not on the type of the test. In the absence of any such markup: - The regular compile test tries to compile a source file, and is deemed a pass if compilation succeeds - The compile-fail test tries to compile a source file, and is deemed a pass if compilation fails Pass is green, fail is yellow.
Yes, the cause may be located elsewhere, in Geometry, Regression, Test, etc.
For a clean run I'm now unable to build the tests due to some undefined references but the problem is most likely located elsewhere.
I think you first need to determine whether the actual compilation of the source file results in success (i.e. object file is produced) or failure (i.e. errors from compiler). For a compile-fail test, the compilation is expected to result in failure. If it does not, it's a problem in test itself or geometry library. If compilation results in failure, but regression matrix shows it as failed test, then something is wrong down the pipeline. -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
Hi Vladimir, Vladimir Prus wrote:
I am fairly sure that dark green vs green depends only on status/explicit-failures-markup.xml, and not on the type of the test. In the absence of any such markup:
- The regular compile test tries to compile a source file, and is deemed a pass if compilation succeeds - The compile-fail test tries to compile a source file, and is deemed a pass if compilation fails
Pass is green, fail is yellow.
Still AFAIU compile-fail should work as expected even if the test wasn't marked as failing in the xml file.
Yes, the cause may be located elsewhere, in Geometry, Regression, Test, etc.
For a clean run I'm now unable to build the tests due to some undefined references but the problem is most likely located elsewhere.
I think you first need to determine whether the actual compilation of the source file results in success (i.e. object file is produced) or failure (i.e. errors from compiler). For a compile-fail test, the compilation is expected to result in failure. If it does not, it's a problem in test itself or geometry library. If compilation results in failure, but regression matrix shows it as failed test, then something is wrong down the pipeline.
Yes, you're right. The test is compiling, and the object file is created. But to detect this I was forced to compile it manually. The message produced by b2 is slightly missleading. On my machine it looks like this: gcc.compile.c++ ..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o "g++" -ftemplate-depth-128 -O0 -fno-inline -Wall -g -std=c++11 -DBOOST_ALL_NO_LIB=1 -DBOOST_CHRONO_DYN_LINK=1 -DBOOST_CHRONO_THREAD_DISABLED -DBOOST_SYSTEM_DYN_LINK=1 -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_TIMER_DYN_LINK=1 -I"..\.." -I"..\..\boost\geometry\extensions\contrib\ttmath" -I"index\test" -I"test" -c -o "..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o" "index\test\rtree\rtree_values_invalid.cpp" ...failed gcc.compile.c++ ..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o... ...removing ..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o So the test fails to compile. The compiler output is not printed, but not in all cases. The last line doesn't really mean anything, it might as well be some cleanup just in case. If the test fails to compile (failure not expected) the compiler output is printed, bjam's output is nearly the same. Just the last line "...removing ..." is not displayed. On the other hand when the compilation failure is expected and the test really fails to compile bjam displays nice (failed-as-expected) instead of "...failed gcc.compile.c++". Would it be possible to display slightly different message when a test expected to fail didn't fail, e.g. (passed-unexpectedly), (unexpected-pass) or something similar? At least not "...failed gcc.compile.c++" because the compilation didn't really fail. Regards, Adam
Adam, On 01/13/2015 03:36 AM, Adam Wulkiewicz wrote:
I think you first need to determine whether the actual compilation of the source file results in success (i.e. object file is produced) or failure (i.e. errors from compiler). For a compile-fail test, the compilation is expected to result in failure. If it does not, it's a problem in test itself or geometry library. If compilation results in failure, but regression matrix shows it as failed test, then something is wrong down the pipeline.
Yes, you're right. The test is compiling, and the object file is created. But to detect this I was forced to compile it manually. The message produced by b2 is slightly missleading. On my machine it looks like this:
gcc.compile.c++ ..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o
"g++" -ftemplate-depth-128 -O0 -fno-inline -Wall -g -std=c++11 -DBOOST_ALL_NO_LIB=1 -DBOOST_CHRONO_DYN_LINK=1 -DBOOST_CHRONO_THREAD_DISABLED -DBOOST_SYSTEM_DYN_LINK=1 -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_TIMER_DYN_LINK=1 -I"..\.." -I"..\..\boost\geometry\extensions\contrib\ttmath" -I"index\test" -I"test" -c -o "..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o" "index\test\rtree\rtree_values_invalid.cpp"
...failed gcc.compile.c++ ..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o... ...removing ..\..\bin.v2\libs\geometry\index\test\rtree\rtree_values_invalid.test\gcc-mingw-4.9.1\debug\rtree_values_invalid.o
So the test fails to compile. The compiler output is not printed, but not in all cases. The last line doesn't really mean anything, it might as well be some cleanup just in case.
If the test fails to compile (failure not expected) the compiler output is printed, bjam's output is nearly the same. Just the last line "...removing ..." is not displayed.
On the other hand when the compilation failure is expected and the test really fails to compile bjam displays nice (failed-as-expected) instead of "...failed gcc.compile.c++".
Would it be possible to display slightly different message when a test expected to fail didn't fail, e.g. (passed-unexpectedly), (unexpected-pass) or something similar?
That's a good suggestion! I will add this to my todo list, but won't be able to address it immediately. Thanks, -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
Vladimir Prus wrote:
On 01/13/2015 03:36 AM, Adam Wulkiewicz wrote:
Would it be possible to display slightly different message when a test expected to fail didn't fail, e.g. (passed-unexpectedly), (unexpected-pass) or something similar?
That's a good suggestion! I will add this to my todo list, but won't be able to address it immediately.
Ok, great. Thanks for your help. Adam
Vladimir Prus wrote:
On 01/13/2015 03:36 AM, Adam Wulkiewicz wrote:
Would it be possible to display slightly different message when a test expected to fail didn't fail, e.g. (passed-unexpectedly), (unexpected-pass) or something similar?
That's a good suggestion! I will add this to my todo list, but won't be able to address it immediately.
When you do that, can you please make sure that b2 | grep failed still displays all failures?
On 01/13/2015 06:29 PM, Peter Dimov wrote:
Vladimir Prus wrote:
On 01/13/2015 03:36 AM, Adam Wulkiewicz wrote:
Would it be possible to display slightly different message when a test > expected to fail didn't fail, e.g. (passed-unexpectedly), (unexpected-pass) or something similar?
That's a good suggestion! I will add this to my todo list, but won't be able to address it immediately.
When you do that, can you please make sure that b2 | grep failed still displays all failures?
Sure. -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
participants (3)
-
Adam Wulkiewicz -
Peter Dimov -
Vladimir Prus