[regression][testing][config] Regression results not displayed.

Hi, Since 2015/04/26 the tests started to dissapear from Geometry summary pages for develop branch. And now the pages aren't even displayed properly, I see only: Error extracting file: No matching files were found. for: http://www.boost.org/development/tests/develop/developer/geometry.html http://www.boost.org/development/tests/develop/developer/geometry-extensions... and on: http://www.boost.org/development/tests/develop/developer/geometry-index.html there are only 2 tests. Furthermore "geometry" and "geometry-extensions" can't be found in the menu. I can see the results for Geometry in the paskages sent to the server. I can also see them in the sanitizer reports sent by Ben. So I'm guessing that the error must be on a later stage of processing, probably in the Regression. Just before the time I saw the issue the first time I added new tests to "geometry-extensions", though it seems wierd that it could affect "geometry" and "geometry-index": https://github.com/boostorg/geometry/commit/9bc8baaa3b3fffc96fbb0abd10a8cb02... https://github.com/boostorg/geometry/commit/07c8b24b89e9fec75c6b7efc7e940328... Geometry is probably the library with the most complex tests structure. We use nested directories and build-project directive in Jamfiles. May this be a part of the problem? AFAIR at that time there were also some changes in the Regression, and the last time something like this happened Config was changed. Could some changes in those modules cause the issue I'm writing about? What may be the reason of such behavior and most importantly do you know how to fix it? Regards, Adam

Adam Wulkiewicz wrote:
Since 2015/04/26 the tests started to dissapear from Geometry summary pages for develop branch. And now the pages aren't even displayed properly, I see only:
Error extracting file: No matching files were found.
<snip>
What may be the reason of such behavior and most importantly do you know how to fix it?
I did more research and I found out that this change in Config (from Apr 24, 2015) is preventing the generation of the proper XML output for the Geometry tests: https://github.com/boostorg/config/commit/d7e258faf5ca45734a9f0349f2a9decedd... It's because when I'm running tests for the the boost revision corresponding to this change and Geometry develop the tests for Geometry aren't listed by b2. If Config isn't tested (e.g. tests are limited to geometry or geometry|math, etc.) everything is ok. Furthermore if boost revision is set to the parent commit then the tests for Geometry develop are listed properly. More info (reproducing and my output) at the end of the email. Do you know why the above change may cause the issue or more importantly how it could be fixed? This isn't the first time something like this is hapening. Before a change in Config affected not only Geometry but also Spirit. Do you have an idea why is this happening and how could this be prevented in the future? E.g. could it be caused by some error in Build? Would it be possible to somehow debug the bjam or building scripts to see why there is a problem? Is it possible to display some additional bjam debugging info which could allow to track the issue? If not then would it be possible to add it? FYI, the tests are present in the output XML but for them (tag "test-log") attributes "revision", "test-run" and "test-program" are not set. So the tests are performed, the log is there in the XML but because they wasn't somehow marked for execution (listed) the attributes are not set. This prevents the Regression's boost-report program to generate summay pages for Geometry. I'm guessing that such situation when the tests are run but can't be found on the list of the tests for output could be somehow detected and reported. Regards, Adam P.S. If you'd like to reproduce it. I'm testing it like this (from BOOST_ROOT): git checkout 4ba573a git submodule update cd libs/geometry git checkout develop after this Boost is set to the failing commit and Geometry to develop. As mentioned above it corresponds to https://github.com/boostorg/config/commit/d7e258faf5ca45734a9f0349f2a9decedd... Then: cd ../.. b2 "--debug-configuration" --dump-tests "--limit-tests=geometry|config" status you can of course break after all of the tests are listed. I see: [some debug output] [listed tests for other libraries] boost-test(LINK) "config/has_atomic_lib" : "libs/config/test/config_info.cpp" boost-test(COMPILE) "config/config_test_c" : "libs/config/test/config_test_c.c" boost-test(RUN) "config/config_test_threaded" : "libs/config/test/config_test.cpp" boost-test(RUN) "config/config_test" : "libs/config/test/config_test.cpp" boost-test(RUN) "config/config_test_no_rtti" : "libs/config/test/config_test.cpp" boost-test(RUN) "config/config_test_no_except" : "libs/config/test/config_test.cpp" boost-test(RUN) "config/config_info" ["always_show_run_output"] : "libs/config/test/config_info.cpp" boost-test(RUN) "config/config_info_threaded" ["always_show_run_output"] : "libs/config/test/config_info.cpp" boost-test(RUN) "config/config_info_no_rtti" ["always_show_run_output"] : "libs/config/test/config_info.cpp" boost-test(RUN) "config/config_info_no_except" ["always_show_run_output"] : "libs/config/test/config_info.cpp" boost-test(RUN) "config/math_info" ["always_show_run_output"] : "libs/config/test/math_info.cpp" boost-test(RUN) "config/abi_test" : "libs/config/test/abi/abi_test.cpp" "libs/config/test/abi/main.cpp" boost-test(RUN) "config/limits_test" : "libs/config/test/limits_test.cpp" boost-test(RUN) "config/config_link_test" : "libs/config/test/link/main.cpp" boost-test(COMPILE_FAIL) "config/test_thread_fail1" : "libs/config/test/threads/test_thread_fail1.cpp" boost-test(COMPILE_FAIL) "config/test_thread_fail2" : "libs/config/test/threads/test_thread_fail2.cpp" boost-test(COMPILE) "config/boost_fallthrough_test" : "libs/config/test/boost_fallthrough_test.cpp" boost-test(COMPILE) "config/helper_macro_test" : "libs/config/test/helper_macro_test.cpp" boost-test(RUN) "config/cstdint_test" : "libs/config/test/cstdint_test.cpp" boost-test(RUN) "config/cstdint_test2" : "libs/config/test/cstdint_test2.cpp" boost-test(COMPILE) "config/cstdint_include_test" : "libs/config/test/cstdint_include_test.cpp" boost-test(RUN) "config/config_build_check" : "libs/config/test/config_build_check.cpp" boost-test(RUN) "geometry/index/varray_old" : "libs/geometry/index/test/varray_old.cpp" boost-test(RUN) "geometry/index/varray" : "libs/geometry/index/test/varray.cpp" - has_atomic_lib builds : no - Clang implicit fallthrough : no - Boost.Config Feature Check: int128 : no - Boost.Config Feature Check: cxx11_constexpr : no - Boost.Config Feature Check: cxx11_user_defined_literals : no ...patience... Those are the only 2 tests for Geometry. Those are the only ones that are currently displayed on the website. Then go to the Boost revision right before the above one and execute the tests again: git checkout 826b356 git submodule update cd libs/geometry git checkout develop cd ../.. b2 "--debug-configuration" --dump-tests "--limit-tests=geometry|config" status Now I can see: [some debug output] - Boost.Config Feature Check: int128 : no (cached) - Boost.Config Feature Check: cxx11_constexpr : no (cached) - Boost.Config Feature Check: cxx11_user_defined_literals : no (cached) [listed tests for other libraries] boost-test(COMPILE) "config/config_test_c" : "libs/config/test/config_test_c.c" boost-test(RUN) "config/config_test_threaded" : "libs/config/test/config_test.cpp" boost-test(RUN) "config/config_test" : "libs/config/test/config_test.cpp" boost-test(RUN) "config/config_test_no_rtti" : "libs/config/test/config_test.cpp" boost-test(RUN) "config/config_test_no_except" : "libs/config/test/config_test.cpp" boost-test(RUN) "config/config_info" ["always_show_run_output"] : "libs/config/test/config_info.cpp" boost-test(RUN) "config/config_info_threaded" ["always_show_run_output"] : "libs/config/test/config_info.cpp" boost-test(RUN) "config/config_info_no_rtti" ["always_show_run_output"] : "libs/config/test/config_info.cpp" boost-test(RUN) "config/config_info_no_except" ["always_show_run_output"] : "libs/config/test/config_info.cpp" boost-test(RUN) "config/math_info" ["always_show_run_output"] : "libs/config/test/math_info.cpp" boost-test(RUN) "config/abi_test" : "libs/config/test/abi/abi_test.cpp" "libs/config/test/abi/main.cpp" boost-test(RUN) "config/limits_test" : "libs/config/test/limits_test.cpp" boost-test(RUN) "config/config_link_test" : "libs/config/test/link/main.cpp" boost-test(COMPILE_FAIL) "config/test_thread_fail1" : "libs/config/test/threads/test_thread_fail1.cpp" boost-test(COMPILE_FAIL) "config/test_thread_fail2" : "libs/config/test/threads/test_thread_fail2.cpp" boost-test(COMPILE) "config/boost_fallthrough_test" : "libs/config/test/boost_fallthrough_test.cpp" boost-test(COMPILE) "config/helper_macro_test" : "libs/config/test/helper_macro_test.cpp" boost-test(RUN) "config/cstdint_test" : "libs/config/test/cstdint_test.cpp" boost-test(RUN) "config/cstdint_test2" : "libs/config/test/cstdint_test2.cpp" boost-test(COMPILE) "config/cstdint_include_test" : "libs/config/test/cstdint_include_test.cpp" boost-test(RUN) "config/config_build_check" : "libs/config/test/config_build_check.cpp" boost-test(RUN) "geometry/index/varray_old" : "libs/geometry/index/test/varray_old.cpp" boost-test(RUN) "geometry/index/varray" : "libs/geometry/index/test/varray.cpp" boost-test(RUN) "config/link_test_test" : "libs/config/test/link/main.cpp" boost-test(RUN) "geometry/access" : "libs/geometry/test/core/access.cpp" boost-test(RUN) "geometry/coordinate_dimension" : "libs/geometry/test/core/coordinate_dimension.cpp" boost-test(RUN) "geometry/coordinate_system" : "libs/geometry/test/core/coordinate_system.cpp" boost-test(RUN) "geometry/coordinate_type" : "libs/geometry/test/core/coordinate_type.cpp" [the rest of the tests for Geometry] ...patience... So AFAIU the issue can be indeed related to the checks for has_atomic_lib.

On 10/05/2015 15:37, Adam Wulkiewicz wrote:
Adam Wulkiewicz wrote:
Since 2015/04/26 the tests started to dissapear from Geometry summary pages for develop branch. And now the pages aren't even displayed properly, I see only:
Error extracting file: No matching files were found.
<snip>
What may be the reason of such behavior and most importantly do you know how to fix it?
I did more research and I found out that this change in Config (from Apr 24, 2015) is preventing the generation of the proper XML output for the Geometry tests:
https://github.com/boostorg/config/commit/d7e258faf5ca45734a9f0349f2a9decedd...
Oh :( I don't see any easy fix, and besides the check-target-builds rule is used all over the place. And at present the whole testing build seems to be broken with: M:/data/boost/boost/libs/predef/test\../check/predef.jam:18: in load *** argument error * rule project.extension ( id : options * : * ) * called with: ( predef check ) * extra argument check M:/data/boost/boost/tools/build/src/build\project.jam:908:see definition of rule 'project.extension' being called M:/data/boost/boost/tools/build/src/kernel\modules.jam:289: in modules.import M:/data/boost/boost/tools/build/src/build\project.jam:1024: in import ..\libs\predef\test\build.jam:7: in modules.load M:/data/boost/boost/tools/build/src/build\project.jam:311: in load-jamfile M:/data/boost/boost/tools/build/src/build\project.jam:64: in load M:/data/boost/boost/tools/build/src/build\project.jam:89: in load-used-projects M:/data/boost/boost/tools/build/src/build\project.jam:75: in load M:/data/boost/boost/tools/build/src/build\project.jam:145: in project.find M:/data/boost/boost/tools/build/src\build-system.jam:535: in load M:\data\boost\boost\tools\build\src/kernel\modules.jam:289: in import M:\data\boost\boost\tools\build\src/kernel/bootstrap.jam:139: in boost-build M:\data\boost\boost\boost-build.jam:17: in module scope Any Boost.Build experts out there? John.

On 10/05/2015 15:37, Adam Wulkiewicz wrote:
Adam Wulkiewicz wrote:
Since 2015/04/26 the tests started to dissapear from Geometry summary pages for develop branch. And now the pages aren't even displayed properly, I see only:
Error extracting file: No matching files were found.
<snip>
What may be the reason of such behavior and most importantly do you know how to fix it?
I did more research and I found out that this change in Config (from Apr 24, 2015) is preventing the generation of the proper XML output for the Geometry tests:
https://github.com/boostorg/config/commit/d7e258faf5ca45734a9f0349f2a9decedd...
I think I may no what the issue is, and have a workaround, I'll look into it tomorrow first thing, John.

On 10/05/2015 15:37, Adam Wulkiewicz wrote:
Adam Wulkiewicz wrote:
Since 2015/04/26 the tests started to dissapear from Geometry summary pages for develop branch. And now the pages aren't even displayed properly, I see only:
Error extracting file: No matching files were found.
<snip>
What may be the reason of such behavior and most importantly do you know how to fix it?
I did more research and I found out that this change in Config (from Apr 24, 2015) is preventing the generation of the proper XML output for the Geometry tests:
https://github.com/boostorg/config/commit/d7e258faf5ca45734a9f0349f2a9decedd...
Can you see if this: https://github.com/boostorg/config/commit/fa954b8b61718dcdfd21504dbacc01cce3... Fixes things? If so I suspect there are some other places where this needs fixing as well, John.

John Maddock wrote:
On 10/05/2015 15:37, Adam Wulkiewicz wrote:
I did more research and I found out that this change in Config (from Apr 24, 2015) is preventing the generation of the proper XML output for the Geometry tests:
https://github.com/boostorg/config/commit/d7e258faf5ca45734a9f0349f2a9decedd...
Can you see if this: https://github.com/boostorg/config/commit/fa954b8b61718dcdfd21504dbacc01cce3...
Fixes things?
It looks ok for me, i.e. the tests are listed. I'm running the tests for Config and Geometry so for a while we'll see if it works.
If so I suspect there are some other places where this needs fixing as well, John.
Thanks for your help John! Adam

John Maddock wrote:
On 10/05/2015 15:37, Adam Wulkiewicz wrote:
Adam Wulkiewicz wrote:
Since 2015/04/26 the tests started to dissapear from Geometry summary pages for develop branch. And now the pages aren't even displayed properly, I see only:
Error extracting file: No matching files were found.
<snip>
What may be the reason of such behavior and most importantly do you know how to fix it?
I did more research and I found out that this change in Config (from Apr 24, 2015) is preventing the generation of the proper XML output for the Geometry tests:
https://github.com/boostorg/config/commit/d7e258faf5ca45734a9f0349f2a9decedd...
Can you see if this: https://github.com/boostorg/config/commit/fa954b8b61718dcdfd21504dbacc01cce3...
Fixes things?
The XML with the results looks good, test-log attributes are set for Geometry tests. And one of the testers already uploaded the results. So it seems that the fix works. Thanks Adam
If so I suspect there are some other places where this needs fixing as well, John.
participants (2)
-
Adam Wulkiewicz
-
John Maddock