[testing] Dissapearing regression results
Hi, I noticed that for the latest runs the tests of Geometry are missing. Surprisingly not all, 2 tests for geometry/index sublib are present. Initially I thought that it may be caused by the recent modularization but I didn't noticed any problems in any of the libraries I checked randomly. Besides, I made some changes in Geometry recently, which AFAIU shouldn't cause any problems, but I guess you may never know. What I changed was: - creation of some files - moving of some files - adding a Jamfile in a directory different than test directories (doc/src/docutils/tools/support_status) - uncommenting tests in file extensions/test/nsphere/Jamfile.v2 So I decided to try running the tests by myself. I tried: 1. Windows + GitBash -> a lot of "can't find xxx file" errors (b2 headers didn't work properly) 2. Windows + GitBash with admin rights -> exactly what can be seen in the tests, bjam.log shows that only 2 tests from Geometry was choosen for testing 3. Windows + cmd with admin rights -> everything is ok (in case you wanted to check it - runner Test2) 4. Linux -> also everything is ok (runner Test) Is it possible that I was lucky and something was fixed yesterday evening (between 2nd and 3rd attempt)? Might my changes cause some problems? E.g. because of some state kept by the runners? In Geometry we have Jamfiles not only in test and xxx/test directories but also in examples, docs and in the main module directory. May this cause some problems? At least for me, on Windows b2 headers now requires admin rights. Otherwise the links just aren't created. Is it because of the replacing hardlinks with symlinks? Another thing is that symlinks aren't properly created for all libraries. AFAIR previously something like this happened after the addition of sublibs file. I don't remember exactly but it was probably caused by the same names of the older tests and newer ones from different directories, so not the same issue? Is there something we could do about it? Regards, Adam
Adam Wulkiewicz wrote:
Hi,
I noticed that for the latest runs the tests of Geometry are missing. Surprisingly not all, 2 tests for geometry/index sublib are present.
Initially I thought that it may be caused by the recent modularization but I didn't noticed any problems in any of the libraries I checked randomly. <snip> Now I see problems in other libraries too. So I decided to try running the tests by myself. I tried: 1. Windows + GitBash -> a lot of "can't find xxx file" errors (b2 headers didn't work properly) 2. Windows + GitBash with admin rights -> exactly what can be seen in the tests, bjam.log shows that only 2 tests from Geometry was choosen for testing 3. Windows + cmd with admin rights -> everything is ok (in case you wanted to check it - runner Test2) 4. Linux -> also everything is ok (runner Test) 3 and 4 was ok because there was only 1 module tested --libraries=geometry. For 2 I tested also core and test.
I tested it locally running (in case someone would like to play with it too): b2 status --dump-tests --debug-configuration and found out that the problem is related with the usage of 'build-project' in Jamfiles. This is not only seen in Geometry, though it's the library that uses this instruction most extensively. E.g. see: http://www.boost.org/development/tests/develop/developer/geometry.html http://www.boost.org/development/tests/develop/developer/math.html http://www.boost.org/development/tests/develop/developer/functional-hash.htm... The projects aren't parsed recursively since ~6.06.2014. What's going on? May it be caused by some invalid Jamfiles added recently for some other libraries? Regards, Adam
Adam Wulkiewicz wrote:
Adam Wulkiewicz wrote:
Hi,
I noticed that for the latest runs the tests of Geometry are missing. Surprisingly not all, 2 tests for geometry/index sublib are present.
Initially I thought that it may be caused by the recent modularization but I didn't noticed any problems in any of the libraries I checked randomly. <snip> Now I see problems in other libraries too. So I decided to try running the tests by myself. I tried: 1. Windows + GitBash -> a lot of "can't find xxx file" errors (b2 headers didn't work properly) 2. Windows + GitBash with admin rights -> exactly what can be seen in the tests, bjam.log shows that only 2 tests from Geometry was choosen for testing 3. Windows + cmd with admin rights -> everything is ok (in case you wanted to check it - runner Test2) 4. Linux -> also everything is ok (runner Test) 3 and 4 was ok because there was only 1 module tested --libraries=geometry. For 2 I tested also core and test.
I tested it locally running (in case someone would like to play with it too): b2 status --dump-tests --debug-configuration
and found out that the problem is related with the usage of 'build-project' in Jamfiles. This is not only seen in Geometry, though it's the library that uses this instruction most extensively.
E.g. see: http://www.boost.org/development/tests/develop/developer/geometry.html http://www.boost.org/development/tests/develop/developer/math.html http://www.boost.org/development/tests/develop/developer/functional-hash.htm...
The projects aren't parsed recursively since ~6.06.2014. What's going on? May it be caused by some invalid Jamfiles added recently for some other libraries?
I isolated the problem. It's in the Config tests. I believe this commit introduces the issue: https://github.com/boostorg/config/commit/dad7bc5b30a6b313208e2b2eefe47aa1f2... It's related to the addition of the Config checks to the test/Jamfile.v2: import ../checks/config : requires ; and [ run config_build_check.cpp : : : [ requires int128 cxx11_constexpr cxx11_user_defined_literals ] ] I'm curious why this addition creates this issue? Could someone explain? And could you please fix it? My attempt would end with removing of 'requires' and this is probably not how it should be fixed. Regards, Adam
Updating topic to better reflect the content... Adam Wulkiewicz wrote:
Adam Wulkiewicz wrote:
Adam Wulkiewicz wrote:
Hi,
I noticed that for the latest runs the tests of Geometry are missing. Surprisingly not all, 2 tests for geometry/index sublib are present.
Initially I thought that it may be caused by the recent modularization but I didn't noticed any problems in any of the libraries I checked randomly. <snip> Now I see problems in other libraries too. So I decided to try running the tests by myself. I tried: 1. Windows + GitBash -> a lot of "can't find xxx file" errors (b2 headers didn't work properly) 2. Windows + GitBash with admin rights -> exactly what can be seen in the tests, bjam.log shows that only 2 tests from Geometry was choosen for testing 3. Windows + cmd with admin rights -> everything is ok (in case you wanted to check it - runner Test2) 4. Linux -> also everything is ok (runner Test) 3 and 4 was ok because there was only 1 module tested --libraries=geometry. For 2 I tested also core and test.
I tested it locally running (in case someone would like to play with it too): b2 status --dump-tests --debug-configuration
and found out that the problem is related with the usage of 'build-project' in Jamfiles. This is not only seen in Geometry, though it's the library that uses this instruction most extensively.
E.g. see: http://www.boost.org/development/tests/develop/developer/geometry.html http://www.boost.org/development/tests/develop/developer/math.html http://www.boost.org/development/tests/develop/developer/functional-hash.htm...
The projects aren't parsed recursively since ~6.06.2014. What's going on? May it be caused by some invalid Jamfiles added recently for some other libraries?
I isolated the problem. It's in the Config tests. I believe this commit introduces the issue: https://github.com/boostorg/config/commit/dad7bc5b30a6b313208e2b2eefe47aa1f2...
It's related to the addition of the Config checks to the test/Jamfile.v2:
import ../checks/config : requires ;
and
[ run config_build_check.cpp : : : [ requires int128 cxx11_constexpr cxx11_user_defined_literals ] ]
I'm curious why this addition creates this issue? Could someone explain?
And could you please fix it? My attempt would end with removing of 'requires' and this is probably not how it should be fixed.
Regards, Adam
[ run config_build_check.cpp : : : [ requires int128 cxx11_constexpr cxx11_user_defined_literals ] ]
I'm curious why this addition creates this issue? Could someone explain?
And could you please fix it? My attempt would end with removing of 'requires' and this is probably not how it should be fixed.
Ugh, well it's a new test that's been added, I don't understand why it would impact on other libraries at all? I can revert the test, but that's not really a solution either? John.
John Maddock wrote:
[ run config_build_check.cpp : : : [ requires int128 cxx11_constexpr cxx11_user_defined_literals ] ]
I'm curious why this addition creates this issue? Could someone explain?
And could you please fix it? My attempt would end with removing of 'requires' and this is probably not how it should be fixed.
Ugh, well it's a new test that's been added, I don't understand why it would impact on other libraries at all? I can revert the test, but that's not really a solution either?
Sorry, I don't know why is this a problem. Maybe someone could look at Build? The 1.56 release date is comming and some of the tests are missing. Are those newly added checks important? Could it be disabled for now, to see if this is indeed the problem? Or maybe could those requirements be checked differently? Regards, Adam
On Mon, Jun 9, 2014 at 6:45 AM, John Maddock
[ run config_build_check.cpp : : : [ requires int128 cxx11_constexpr
cxx11_user_defined_literals ] ]
I'm curious why this addition creates this issue? Could someone explain?
And could you please fix it? My attempt would end with removing of 'requires' and this is probably not how it should be fixed.
Ugh, well it's a new test that's been added, I don't understand why it would impact on other libraries at all? I can revert the test, but that's not really a solution either?
John, Have you confirmed that it is actually this commit that is causing the failure? Like you, " I don't understand why it would impact on other libraries at all?" --Beman
John,
Have you confirmed that it is actually this commit that is causing the failure?
Like you, " I don't understand why it would impact on other libraries at all?"
No, and I'm unable to run the tests in /status/ either with or without the new config test present, I get some typically inscrutable Boost.Build errors: M:/data/boost/boost/tools/build/src/build\property.jam:740: in class@property-map.find from module object(property-map)@255 error: Ambiguous key M:/data/boost/boost/tools/build/src/build\type.jam:333: in generated-target-ps from module type M:/data/boost/boost/tools/build/src/build\type.jam:270: in type.generated-target -suffix from module type M:/data/boost/boost/tools/build/src/build\virtual-target.jam:505: in virtual-target.add-prefix-and-suffix from module virtual-target M:/data/boost/boost/tools/build/src/build\virtual-target.jam:464: in _adjust-name from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/build\virtual-target.jam:253: in abstract-file-target.__init__ from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/build\virtual-target.jam:557: in class@file-target.__init__ from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/kernel\class.jam:90: in class.new from module class How do I checkout the same tree as is used for testing? John.
John Maddock wrote:
John,
Have you confirmed that it is actually this commit that is causing the failure?
Like you, " I don't understand why it would impact on other libraries at all?"
No, and I'm unable to run the tests in /status/ either with or without the new config test present, I get some typically inscrutable Boost.Build errors:
M:/data/boost/boost/tools/build/src/build\property.jam:740: in class@property-map.find from module object(property-map)@255 error: Ambiguous key M:/data/boost/boost/tools/build/src/build\type.jam:333: in generated-target-ps from module type M:/data/boost/boost/tools/build/src/build\type.jam:270: in type.generated-target -suffix from module type M:/data/boost/boost/tools/build/src/build\virtual-target.jam:505: in virtual-target.add-prefix-and-suffix from module virtual-target M:/data/boost/boost/tools/build/src/build\virtual-target.jam:464: in _adjust-name from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/build\virtual-target.jam:253: in abstract-file-target.__init__ from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/build\virtual-target.jam:557: in class@file-target.__init__ from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/kernel\class.jam:90: in class.new from module class
How do I checkout the same tree as is used for testing?
During testing bjam is built and run like this: # Starting tests ("D:\lib\test-boost3\tools_bb\src\engine\bin.ntx86\bjam.exe" "-sBOOST_BUILD_PATH=D:\lib\test-boost3;D:\lib\test-boost3\tools_bb\src" "-sBOOST_ROOT=D:\lib\test-boost3\boost_root" "--bo ost=D:\lib\test-boost3\boost_root" "--boost-build=D:\lib\test-boost3\tools_bb\src" "--debug-configuration" -l300 toolset=msvc-10.0 -d2 preserve-test-targets=off --dump-tests -m64 "--build-dir=D:\lib \test-boost3\results" >>"D:\lib\test-boost3\results\bjam.log" 2>&1)... AFAIU if you just run (from BOOST_ROOT/status): b2 --dump-tests you should get similar results. So e.g. after running: b2 --dump-test | grep 'geometry' you should get the list of all of the tests which will be run for Geometry. Currently only 2. Regards, Adam
No, and I'm unable to run the tests in /status/ either with or without the new config test present, I get some typically inscrutable Boost.Build errors:
M:/data/boost/boost/tools/build/src/build\property.jam:740: in class@property-map.find from module object(property-map)@255 error: Ambiguous key M:/data/boost/boost/tools/build/src/build\type.jam:333: in generated-target-ps from module type M:/data/boost/boost/tools/build/src/build\type.jam:270: in type.generated-target -suffix from module type M:/data/boost/boost/tools/build/src/build\virtual-target.jam:505: in virtual-target.add-prefix-and-suffix from module virtual-target M:/data/boost/boost/tools/build/src/build\virtual-target.jam:464: in _adjust-name from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/build\virtual-target.jam:253: in abstract-file-target.__init__ from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/build\virtual-target.jam:557: in class@file-target.__init__ from module object(file-target)@98264 M:/data/boost/boost/tools/build/src/kernel\class.jam:90: in class.new from module class
How do I checkout the same tree as is used for testing?
During testing bjam is built and run like this:
# Starting tests ("D:\lib\test-boost3\tools_bb\src\engine\bin.ntx86\bjam.exe" "-sBOOST_BUILD_PATH=D:\lib\test-boost3;D:\lib\test-boost3\tools_bb\src" "-sBOOST_ROOT=D:\lib\test-boost3\boost_root" "--bo ost=D:\lib\test-boost3\boost_root" "--boost-build=D:\lib\test-boost3\tools_bb\src" "--debug-configuration" -l300 toolset=msvc-10.0 -d2 preserve-test-targets=off --dump-tests -m64 "--build-dir=D:\lib \test-boost3\results" >>"D:\lib\test-boost3\results\bjam.log" 2>&1)...
AFAIU if you just run (from BOOST_ROOT/status):
b2 --dump-tests
you should get similar results. So e.g. after running:
b2 --dump-test | grep 'geometry'
you should get the list of all of the tests which will be run for Geometry. Currently only 2.
OK, I can reproduce that, although the test run still terminates with the error messages I reported above - which is to say nothing is actually built even with the recent Config changes removed. I think we need an expert... John.
participants (3)
-
Adam Wulkiewicz -
Beman Dawes -
John Maddock