Help with understanding regression test results

newer
[ANN] CppCon 2014 program available

Andrzej Krzemienski

23 Jun 2014 23 Jun '14

7:46 a.m.

Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional: http://www.boost.org/development/tests/develop/developer/optional.html When I try to inspect the results they are either empty or refer to the places in the code that do not exist anymore in branch 'develop', as though the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is broken with the testing framework. Anyone can help me with figuring out what to do? Regards, &rzej

Show replies by date

Andrey Semashev

23 Jun 23 Jun

9:40 a.m.

On Monday 23 June 2014 09:46:56 Andrzej Krzemienski wrote:

...

Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional: http://www.boost.org/development/tests/develop/developer/optional.html

When I try to inspect the results they are either empty or refer to the places in the code that do not exist anymore in branch 'develop', as though the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is broken with the testing framework. Anyone can help me with figuring out what to do?

Which tests do you mean?

Andrzej Krzemienski

10:32 a.m.

2014-06-23 11:40 GMT+02:00 Andrey Semashev :

...

...
Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional: http://www.boost.org/development/tests/develop/developer/optional.html

When I try to inspect the results they are either empty or refer to the places in the code that do not exist anymore in branch 'develop', as

On Monday 23 June 2014 09:46:56 Andrzej Krzemienski wrote: though

...
the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is broken with the testing framework. Anyone can help me with figuring out what to do?

Which tests do you mean?

This one, for instance: http://www.boost.org/development/tests/develop/developer/output/teeks99-03e-... The error message quotes a piece of implementation: value_or ( U const& v ) const { return this->is_initialized() ? get() : (v); } But there exists no such code in the current implementation in branch 'develop'. I am using no ternary operators (I used to in the previous commits). The test's date is that of today. My last commit was on Friday. Regards, &rzej

Andrey Semashev

11:53 a.m.

On Monday 23 June 2014 12:32:31 Andrzej Krzemienski wrote:

...

2014-06-23 11:40 GMT+02:00 Andrey Semashev :

...
On Monday 23 June 2014 09:46:56 Andrzej Krzemienski wrote:

...
Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional: http://www.boost.org/development/tests/develop/developer/optional.html

When I try to inspect the results they are either empty or refer to the places in the code that do not exist anymore in branch 'develop', as

though

...
the test environments were using partial results of GIT commits. But the test dates are after the last commit, so it looks like something is

broken

...
with the testing framework. Anyone can help me with figuring out what to do?

Which tests do you mean?

This one, for instance: http://www.boost.org/development/tests/develop/developer/output/teeks99-03e-> Ubuntu12-04-64-boost-bin-v2-libs-optional-test-optional_test_value_access-te st-gcc-4-8-debug.html

The error message quotes a piece of implementation:

value_or ( U const& v ) const { return this->is_initialized() ? get() : (v); }

But there exists no such code in the current implementation in branch 'develop'. I am using no ternary operators (I used to in the previous commits). The test's date is that of today. My last commit was on Friday.

Yes, looks like a problem. Maybe the checkout failed at some point for the tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.

Andrzej Krzemienski

11:57 a.m.

2014-06-23 13:53 GMT+02:00 Andrey Semashev :

...

...
2014-06-23 11:40 GMT+02:00 Andrey Semashev :

...
On Monday 23 June 2014 09:46:56 Andrzej Krzemienski wrote:

...
Hi, Recently I observed lots of test cases showing as failed in regression tests on branch 'develop' in Boost.Optional:

http://www.boost.org/development/tests/develop/developer/optional.html

...
When I try to inspect the results they are either empty or refer to

...
...
...
places in the code that do not exist anymore in branch 'develop', as

though

...
the test environments were using partial results of GIT commits. But

On Monday 23 June 2014 12:32:31 Andrzej Krzemienski wrote: the the

...
...
...
test dates are after the last commit, so it looks like something is

broken

...
with the testing framework. Anyone can help me with figuring out what to do?

Which tests do you mean?

This one, for instance:

http://www.boost.org/development/tests/develop/developer/output/teeks99-03e-> Ubuntu12-04-64-boost-bin-v2-libs-optional-test-optional_test_value_access-te

...
st-gcc-4-8-debug.html

The error message quotes a piece of implementation:

value_or ( U const& v ) const { return this->is_initialized() ? get() : (v); }

But there exists no such code in the current implementation in branch 'develop'. I am using no ternary operators (I used to in the previous commits). The test's date is that of today. My last commit was on Friday.

Yes, looks like a problem. Maybe the checkout failed at some point for the tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.

What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted? regards, &rzej

Andrey Semashev

12:06 p.m.

On Monday 23 June 2014 13:57:50 Andrzej Krzemienski wrote:

...

2014-06-23 13:53 GMT+02:00 Andrey Semashev :

...
Yes, looks like a problem. Maybe the checkout failed at some point for the tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.

What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted?

Yes, if the contact information is available in the tester description (the link in the test results table header). E.g. for your tester the information is: http://www.boost.org/development/tests/develop/teeks99-03e-Ubuntu12-04-64.ht... Tom is usually reading this list, I hope he's reading this.

Tom Kent

1:36 p.m.

On Mon, Jun 23, 2014 at 7:06 AM, Andrey Semashev wrote:

...

...
2014-06-23 13:53 GMT+02:00 Andrey Semashev :

...
Yes, looks like a problem. Maybe the checkout failed at some point for

On Monday 23 June 2014 13:57:50 Andrzej Krzemienski wrote: the

...
...
tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If the update scripts ignore such errors then that's what may have happened.

What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted?

Yes, if the contact information is available in the tester description (the link in the test results table header). E.g. for your tester the information is:

http://www.boost.org/development/tests/develop/teeks99-03e-Ubuntu12-04-64.ht...

Tom is usually reading this list, I hope he's reading this.

I'm not sure what is going on...that commit "c12eaea160fa97a0b6c503eb8f7fdba938f3126f" was only a few hours old, and was the most recent at the time that the run you linked started (4AM). I didn't think it was possible to get a partial checkout, but who knows. Because of space issues on the VMs that are running, I don't keep anything between runs, so I can't actually go and look at the repo that was used for the test in question. The next run that should be complete is teeks99-03g. It looks like it has the correct code at line 1073 (am I reading that correctly?). Once that one comes up, we'll see if everything has caught up. Also, I verified that teeks99-04a just started up and there were no checkout errors that I could see during the start up, so that one might be worth checking in a couple hours as well. Tom

Andrzej Krzemienski

2:45 p.m.

2014-06-23 15:36 GMT+02:00 Tom Kent :

...

On Mon, Jun 23, 2014 at 7:06 AM, Andrey Semashev < andrey.semashev@gmail.com> wrote:

...
...
2014-06-23 13:53 GMT+02:00 Andrey Semashev
...
Yes, looks like a problem. Maybe the checkout failed at some point

for

...
...
tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If

On Monday 23 June 2014 13:57:50 Andrzej Krzemienski wrote: the the

...
...
update scripts ignore such errors then that's what may have happened.

What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted?

Yes, if the contact information is available in the tester description (the link in the test results table header). E.g. for your tester the information is:

http://www.boost.org/development/tests/develop/teeks99-03e-Ubuntu12-04-64.ht...

...
Tom is usually reading this list, I hope he's reading this.

I'm not sure what is going on...that commit "c12eaea160fa97a0b6c503eb8f7fdba938f3126f" was only a few hours old, and was the most recent at the time that the run you linked started (4AM).

I didn't think it was possible to get a partial checkout, but who knows. Because of space issues on the VMs that are running, I don't keep anything between runs, so I can't actually go and look at the repo that was used for the test in question.

The next run that should be complete is teeks99-03g. It looks like it has the correct code at line 1073 (am I reading that correctly?). Once that one comes up, we'll see if everything has caught up.

Currently on branch 'develop' when you look at GitHub: https://github.com/boostorg/optional/blob/develop/include/boost/optional/opt... line 1073 is in a middle of an if-statement. Tehere is no ternary operator?: involved when calling function value_or. Latest commit (4cbb67e5059b8771768dd6598670fe723ca50475) was done three days ago.

...

Also, I verified that teeks99-04a just started up and there were no checkout errors that I could see during the start up, so that one might be worth checking in a couple hours as well.

Ok, I'll do that. Thanks. &rzej

Andrzej Krzemienski

30 Jun 30 Jun

9:43 a.m.

2014-06-23 15:36 GMT+02:00 Tom Kent :

...

On Mon, Jun 23, 2014 at 7:06 AM, Andrey Semashev < andrey.semashev@gmail.com> wrote:

...
...
2014-06-23 13:53 GMT+02:00 Andrey Semashev
...
Yes, looks like a problem. Maybe the checkout failed at some point

for

...
...
tester, and Boost.Optional submodule was left in outdated state. I sometimes get timeouts from GitHub when I update Boost tree, I think in such cases the tree is left in some semi-updated state; I didn't check though. If

On Monday 23 June 2014 13:57:50 Andrzej Krzemienski wrote: the the

...
...
update scripts ignore such errors then that's what may have happened.

What's the procedure in such cases? Can the owners of the machines/environments /scripts be contacted?

Yes, if the contact information is available in the tester description (the link in the test results table header). E.g. for your tester the information is:

http://www.boost.org/development/tests/develop/teeks99-03e-Ubuntu12-04-64.ht...

...
Tom is usually reading this list, I hope he's reading this.

I'm not sure what is going on...that commit "c12eaea160fa97a0b6c503eb8f7fdba938f3126f" was only a few hours old, and was the most recent at the time that the run you linked started (4AM).

I didn't think it was possible to get a partial checkout, but who knows. Because of space issues on the VMs that are running, I don't keep anything between runs, so I can't actually go and look at the repo that was used for the test in question.

The next run that should be complete is teeks99-03g. It looks like it has the correct code at line 1073 (am I reading that correctly?). Once that one comes up, we'll see if everything has caught up.

Also, I verified that teeks99-04a just started up and there were no checkout errors that I could see during the start up, so that one might be worth checking in a couple hours as well.

7 days after, teeks99-03g and teeks99-04a still fail and error messages refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional. Regards, &rzej

John Maddock

10:29 a.m.

...

7 days after, teeks99-03g and teeks99-04a still fail and error messages refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.

Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well. John.

Andrzej Krzemienski

11:20 a.m.

2014-06-30 12:29 GMT+02:00 John Maddock :

...

7 days after, teeks99-03g and teeks99-04a still fail and error messages

...
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.

Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.

I am not sure if this is related, but this is the response from Niklas, who is hosting the regression on QNX: The symbolic links in $BOOST_ROOT/boost created by "bjam headers" before

...

the build starts point to the wrong files:

$ ls -l /extra/boost_regression/x86/boost_root/boost/optional* lrwxrwxrwx 1 boost users 42 Jun 30 07:10 /extra/boost_regression/x86/boost_root/boost/optional.hpp -> .../libs/convert/include/boost/optional.hpp

/extra/boost_regression/x86/boost_root/boost/optional: total 4 lrwxrwxrwx 1 boost users 65 Jun 30 07:10 bad_optional_access.hpp -> .../../libs/convert/include/boost/optional/bad_optional_access.hpp lrwxrwxrwx 1 boost users 54 Jun 30 07:10 optional.hpp -> .../../libs/convert/include/boost/optional/optional.hpp lrwxrwxrwx 1 boost users 58 Jun 30 07:10 optional_fwd.hpp -> .../../libs/convert/include/boost/optional/optional_fwd.hpp lrwxrwxrwx 1 boost users 57 Jun 30 07:10 optional_io.hpp -> .../../libs/convert/include/boost/optional/optional_io.hpp

Can anyone help me understand where it is controlled which links are pointed to which files when I run 'b2 headers'? Regards, &rzej

Tom Kent

2:54 p.m.

On Mon, Jun 30, 2014 at 6:20 AM, Andrzej Krzemienski wrote:

...

2014-06-30 12:29 GMT+02:00 John Maddock :

...
...
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different

7 days after, teeks99-03g and teeks99-04a still fail and error messages source

...
than https://github.com/boostorg/optional.

Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.

I am not sure if this is related, but this is the response from Niklas, who is hosting the regression on QNX:

The symbolic links in $BOOST_ROOT/boost created by "bjam headers" before

...
the build starts point to the wrong files:

$ ls -l /extra/boost_regression/x86/boost_root/boost/optional* lrwxrwxrwx 1 boost users 42 Jun 30 07:10 /extra/boost_regression/x86/boost_root/boost/optional.hpp -> .../libs/convert/include/boost/optional.hpp

/extra/boost_regression/x86/boost_root/boost/optional: total 4 lrwxrwxrwx 1 boost users 65 Jun 30 07:10 bad_optional_access.hpp -> .../../libs/convert/include/boost/optional/bad_optional_access.hpp lrwxrwxrwx 1 boost users 54 Jun 30 07:10 optional.hpp -> .../../libs/convert/include/boost/optional/optional.hpp lrwxrwxrwx 1 boost users 58 Jun 30 07:10 optional_fwd.hpp -> .../../libs/convert/include/boost/optional/optional_fwd.hpp lrwxrwxrwx 1 boost users 57 Jun 30 07:10 optional_io.hpp -> .../../libs/convert/include/boost/optional/optional_io.hpp

Can anyone help me understand where it is controlled which links are pointed to which files when I run 'b2 headers'?

I show the same thing on mine...all the symlinks under boost point to a copy of the same files that reside in the convert repository: https://github.com/boostorg/convert/tree/develop/include/boost/optional Tom

John Maddock

6:13 p.m.

New subject: [convert] Help with understanding regression test results

...

I show the same thing on mine...all the symlinks under boost point to a copy of the same files that reside in the convert repository: https://github.com/boostorg/convert/tree/develop/include/boost/optional

This is bad, looks like there are two copies of the boost/optional/ tree in Git: * The correct one under optional. * What's described as a "local copy" under convert. But you just can't do that, ever. The convert lib needs to remove it's copies from both develop and master otherwise the tree is completely broken. John.

John Maddock

6:44 p.m.

New subject: [convert] Help with understanding regression test results

...

This is bad, looks like there are two copies of the boost/optional/ tree in Git:

* The correct one under optional. * What's described as a "local copy" under convert.

But you just can't do that, ever. The convert lib needs to remove it's copies from both develop and master otherwise the tree is completely broken.

There are also a bunch of duplicated headers such as boost/none.hpp which have to go too. John.

Vladimir Batov

9:59 p.m.

New subject: [convert] Help with understanding regression test results

On 07/01/2014 04:13 AM, John Maddock wrote:

...

...
I show the same thing on mine...all the symlinks under boost point to a copy of the same files that reside in the convert repository: https://github.com/boostorg/convert/tree/develop/include/boost/optional

This is bad, looks like there are two copies of the boost/optional/ tree in Git:

* The correct one under optional. * What's described as a "local copy" under convert.

But you just can't do that, ever. The convert lib needs to remove it's copies from both develop and master otherwise the tree is completely broken.

The duplicate "optional" code has been removed from "convert". Apologies. V.

John Maddock

1 Jul 1 Jul

4:22 p.m.

New subject: [convert] Help with understanding regression test results

...

...
But you just can't do that, ever. The convert lib needs to remove it's copies from both develop and master otherwise the tree is completely broken.

The duplicate "optional" code has been removed from "convert". Apologies.

Thanks! John.

Tom Kent

30 Jun 30 Jun

2:57 p.m.

On Mon, Jun 30, 2014 at 5:29 AM, John Maddock wrote:

...

7 days after, teeks99-03g and teeks99-04a still fail and error messages

...
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.

Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.

Which files are out of date? All the failures I saw referenced a pch.obj...does that depend on changes to bjam?

John Maddock

4:28 p.m.

...

...
7 days after, teeks99-03g and teeks99-04a still fail and error messages

...
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.

Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.

Which files are out of date? All the failures I saw referenced a pch.obj...does that depend on changes to bjam?

It depends on this change to the test Jamfile which disables pch support for now: https://github.com/boostorg/math/commit/b6081dc422b6067c6df28ead64336c1f376c... John.

Tom Kent

5:43 p.m.

The runner teeks99-01a-win2008-64on64 isn't using '--remove-test-targets', so it shouldn't be affected by that change? On Mon, Jun 30, 2014 at 11:28 AM, John Maddock wrote:

...

7 days after, teeks99-03g and teeks99-04a still fail and error messages

...
...
...
refer to the code that has not been in 'develop' for a long while. There are more environments that report references to a no-longer existent code. It appears as though the sources are downloaded from the different source than https://github.com/boostorg/optional.

Just checked, and Math has out-of-date failures from teeks99-01a-win2008-64on64 as well.

Which files are out of date? All the failures I saw referenced a pch.obj...does that depend on changes to bjam?

It depends on this change to the test Jamfile which disables pch support for now: https://github.com/boostorg/math/commit/ b6081dc422b6067c6df28ead64336c1f376ccb53

John.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/ mailman/listinfo.cgi/boost

John Maddock

6:08 p.m.

...

The runner teeks99-01a-win2008-64on64 isn't using '--remove-test-targets', so it shouldn't be affected by that change?

Well that's very strange then, the next question then would be how come those object files are missing: http://www.boost.org/development/tests/develop/developer/output/teeks99-01a-... Thanks for looking into this, John.

3803

Age (days ago)

3811

Last active (days ago)

List overview

Download

19 comments

5 participants

participants (5)

Andrey Semashev
Andrzej Krzemienski
John Maddock
Tom Kent
Vladimir Batov