[contract] help marking regression test expected failures

newer
Adapting boost::grid_graph vertex...

older
path length too long for 64 bits...

Lorenzo Caminiti

24 Oct 2017 24 Oct '17

11:16 p.m.

Hello all, I need some help with marking some Boost.Contract regression tests as expected failures. I don't understand why my explicit-failures-markup.xml: https://github.com/boostorg/boost/blob/develop/status/explicit-failures-mark... Doesn't math the expected failures I see on the regression report: http://www.boost.org/development/tests/develop/developer/contract.html For example, the following from explicit-failures-markup.xml (around line 887) should marked old-if_copyable test as an expected failure on msvc-11.0: ... <mark-expected-failures> <test name="old-if_copyable"/> <test name="old-if_copyable_macro"/> <toolset name="gcc-gnu-4.7.2"/> <toolset name="gcc-gnu-4.7~c++11"/> <toolset name="gcc-mngw-gnu-4.6c+"/> <toolset name="gcc-mngw-gnu-4.7c+"/> <toolset name="msvc-10.0"/> <toolset name="msvc-11.0"/> <note author="Lorenzo Caminiti"> Test fails because this complier version does not properly implement SFINAE causing incorrect errors on substituion failures for private and protected members (this seems to be fixed in GCC 4.8 and MSVC 12.0). </note> </mark-expected-failures> ... But the old-if_copyable test still shows up as a failure on the regression test report: http://www.boost.org/development/tests/develop/developer/output/teeks99-08-d... Any idea what I am doing wrong? Thank you. --Lorenzo

Show replies by date

Joaquin M López Muñoz

25 Oct 25 Oct

7:09 a.m.

El 25/10/2017 a las 1:16, Lorenzo Caminiti via Boost escribió:

...

Hello all,

I need some help with marking some Boost.Contract regression tests as expected failures.

I don't understand why my explicit-failures-markup.xml: https://github.com/boostorg/boost/blob/develop/status/explicit-failures-mark...

Doesn't math the expected failures I see on the regression report: http://www.boost.org/development/tests/develop/developer/contract.html

Your changes haven been added to develop but not merged to the release version of explicit-failures-markup.xml yet. Joaquín M López Muñoz

Edward Diener

1:38 p.m.

On 10/25/2017 3:09 AM, Joaquin M López Muñoz via Boost wrote:

...

El 25/10/2017 a las 1:16, Lorenzo Caminiti via Boost escribió:

...
Hello all,

I need some help with marking some Boost.Contract regression tests as expected failures.

I don't understand why my explicit-failures-markup.xml: https://github.com/boostorg/boost/blob/develop/status/explicit-failures-mark...

Doesn't math the expected failures I see on the regression report: http://www.boost.org/development/tests/develop/developer/contract.html

Your changes haven been added to develop but not merged to the release version of explicit-failures-markup.xml yet.

Why would not an explicit-failures-markup.xml for 'develop' be applied to 'develop' regression testing ?

...

Joaquín M López Muñoz

Lorenzo Caminiti

2:56 p.m.

On Wed, Oct 25, 2017 at 6:38 AM, Edward Diener via Boost wrote:

...

On 10/25/2017 3:09 AM, Joaquin M López Muñoz via Boost wrote:

...
El 25/10/2017 a las 1:16, Lorenzo Caminiti via Boost escribió:

...
Hello all,

I need some help with marking some Boost.Contract regression tests as expected failures.

I don't understand why my explicit-failures-markup.xml:

https://github.com/boostorg/boost/blob/develop/status/explicit-failures-mark...

Doesn't math the expected failures I see on the regression report: http://www.boost.org/development/tests/develop/developer/contract.html

Your changes haven been added to develop but not merged to the release version of explicit-failures-markup.xml yet.

Why would not an explicit-failures-markup.xml for 'develop' be applied to 'develop' regression testing ?

Correct. I am only looking at develop regression testing so the fact that explicit-failures-markup.xml is not merged to master/release is not the problem. Does anyone know what the issue might be? Maybe I am misspelling the compiler names, but I checked them on the regression test report page... I don't know... Thanks. --Lorenzo

Joaquin M López Muñoz

3:07 p.m.

El 25/10/2017 a las 16:56, Lorenzo Caminiti via Boost escribió:

...

...
On 10/25/2017 3:09 AM, Joaquin M López Muñoz via Boost wrote:

...
El 25/10/2017 a las 1:16, Lorenzo Caminiti via Boost escribió:

...
Hello all,

I need some help with marking some Boost.Contract regression tests as expected failures.

I don't understand why my explicit-failures-markup.xml:

https://github.com/boostorg/boost/blob/develop/status/explicit-failures-mark...

Doesn't math the expected failures I see on the regression report: http://www.boost.org/development/tests/develop/developer/contract.html

Your changes haven been added to develop but not merged to the release version of explicit-failures-markup.xml yet.

Why would not an explicit-failures-markup.xml for 'develop' be applied to 'develop' regression testing ? Correct. I am only looking at develop regression testing so the fact

On Wed, Oct 25, 2017 at 6:38 AM, Edward Diener via Boost wrote: that explicit-failures-markup.xml is not merged to master/release is not the problem.

I *think* the xml version from master is used regardless of whether the reports corresponds to develop or master. Joaquín M López Muñoz

Lorenzo Caminiti

8:21 p.m.

On Wed, Oct 25, 2017 at 8:07 AM, Joaquin M López Muñoz via Boost wrote:

...

El 25/10/2017 a las 16:56, Lorenzo Caminiti via Boost escribió:

...
On Wed, Oct 25, 2017 at 6:38 AM, Edward Diener via Boost wrote:

...
On 10/25/2017 3:09 AM, Joaquin M López Muñoz via Boost wrote:

...
El 25/10/2017 a las 1:16, Lorenzo Caminiti via Boost escribió:

...
https://github.com/boostorg/boost/blob/develop/status/explicit-failures-mark...

Doesn't math the expected failures I see on the regression report: http://www.boost.org/development/tests/develop/developer/contract.html

Your changes haven been added to develop but not merged to the release version of explicit-failures-markup.xml yet.

Why would not an explicit-failures-markup.xml for 'develop' be applied to 'develop' regression testing ?

Correct. I am only looking at develop regression testing so the fact that explicit-failures-markup.xml is not merged to master/release is not the problem.

I *think* the xml version from master is used regardless of whether the reports corresponds to develop or master.

You are right! I just checked and the expected failures showing up on develop regression test report match 100% the markups of explicit-failures-markup.xml from *master* (instead of using the markups from develop). Who can fix this? Making sure that explicit-failures-markup.xml from develop (not master) is used to markup the develop's regression tests. Thank you. --Lorenzo

Edward Diener

8:26 p.m.

On 10/25/2017 4:21 PM, Lorenzo Caminiti via Boost wrote:

...

On Wed, Oct 25, 2017 at 8:07 AM, Joaquin M López Muñoz via Boost wrote:

...
El 25/10/2017 a las 16:56, Lorenzo Caminiti via Boost escribió:

...
On Wed, Oct 25, 2017 at 6:38 AM, Edward Diener via Boost wrote:

...
On 10/25/2017 3:09 AM, Joaquin M López Muñoz via Boost wrote:

...
El 25/10/2017 a las 1:16, Lorenzo Caminiti via Boost escribió:

...
https://github.com/boostorg/boost/blob/develop/status/explicit-failures-mark...

Doesn't math the expected failures I see on the regression report: http://www.boost.org/development/tests/develop/developer/contract.html

Your changes haven been added to develop but not merged to the release version of explicit-failures-markup.xml yet.

Why would not an explicit-failures-markup.xml for 'develop' be applied to 'develop' regression testing ?

Correct. I am only looking at develop regression testing so the fact that explicit-failures-markup.xml is not merged to master/release is not the problem.

I *think* the xml version from master is used regardless of whether the reports corresponds to develop or master.

You are right! I just checked and the expected failures showing up on develop regression test report match 100% the markups of explicit-failures-markup.xml from *master* (instead of using the markups from develop).

Who can fix this? Making sure that explicit-failures-markup.xml from develop (not master) is used to markup the develop's regression tests.

The more general fix is to have the 'develop' regression tests use 'develop' everywhere and have the 'master' regression tests use 'master' everywhere.

...

Thank you. --Lorenzo

Lorenzo Caminiti

26 Oct 26 Oct

2:36 a.m.

On Wed, Oct 25, 2017 at 1:26 PM, Edward Diener via Boost wrote:

...

On 10/25/2017 4:21 PM, Lorenzo Caminiti via Boost wrote:

...
On Wed, Oct 25, 2017 at 8:07 AM, Joaquin M López Muñoz via Boost wrote:

...
El 25/10/2017 a las 16:56, Lorenzo Caminiti via Boost escribió:

...
On Wed, Oct 25, 2017 at 6:38 AM, Edward Diener via Boost wrote:

...
On 10/25/2017 3:09 AM, Joaquin M López Muñoz via Boost wrote:

...
El 25/10/2017 a las 1:16, Lorenzo Caminiti via Boost escribió: > > > > https://github.com/boostorg/boost/blob/develop/status/explicit-failures-mark... > > Doesn't math the expected failures I see on the regression report: > > http://www.boost.org/development/tests/develop/developer/contract.html

Your changes haven been added to develop but not merged to the release version of explicit-failures-markup.xml yet.

Why would not an explicit-failures-markup.xml for 'develop' be applied to 'develop' regression testing ?

Correct. I am only looking at develop regression testing so the fact that explicit-failures-markup.xml is not merged to master/release is not the problem.

I *think* the xml version from master is used regardless of whether the reports corresponds to develop or master.

You are right! I just checked and the expected failures showing up on develop regression test report match 100% the markups of explicit-failures-markup.xml from *master* (instead of using the markups from develop).

Who can fix this? Making sure that explicit-failures-markup.xml from develop (not master) is used to markup the develop's regression tests.

The more general fix is to have the 'develop' regression tests use 'develop' everywhere and have the 'master' regression tests use 'master' everywhere.

Agreed. Someone suggested to me that what maybe needs to be done is here: https://github.com/boostorg/regression/blob/ci/test-reports/ci_build_results... add the line (cd ${boost} && git checkout origin/${1} -- status/explicit-failures-markup.xml) but for that to work, here https://github.com/boostorg/regression/blob/ci/test-reports/ci_build_results... --depth 1 must be removed as the above doesn't work on a shallow clone. However, neither who suggested these changes to me neither I feel confident doing these changes because we've never touched the testing infrastructure... Anyone that can check if the changes above are correct and will make develop regression test reports use the develop branch instead of master? Thanks. --Lorenzo

Peter Dimov

25 Oct 25 Oct

9:29 p.m.

Lorenzo Caminiti wrote:

...

Who can fix this? Making sure that explicit-failures-markup.xml from develop (not master) is used to markup the develop's regression tests.

What I think needs to be done is here: https://github.com/boostorg/regression/blob/ci/test-reports/ci_build_results... add the line (cd ${boost} && git checkout origin/${1} -- status/explicit-failures-markup.xml) but for that to work, here https://github.com/boostorg/regression/blob/ci/test-reports/ci_build_results... --depth 1 must be removed as the above doesn't work on a shallow clone. I don't feel particularly confident doing this change though; I've never touched the testing infrastructure and shell scripts aren't my primary area of expertise. As an alternative, you could just merge your markup to master. :-)

Joaquin M López Muñoz

26 Oct 26 Oct

7:30 a.m.

El 25/10/2017 a las 23:29, Peter Dimov via Boost escribió:

...

Lorenzo Caminiti wrote:

...
Who can fix this? Making sure that explicit-failures-markup.xml from develop (not master) is used to markup the develop's regression tests.

What I think needs to be done is here:

https://github.com/boostorg/regression/blob/ci/test-reports/ci_build_results...

add the line

(cd ${boost} && git checkout origin/${1} -- status/explicit-failures-markup.xml)

but for that to work, here

https://github.com/boostorg/regression/blob/ci/test-reports/ci_build_results...

--depth 1 must be removed as the above doesn't work on a shallow clone.

I don't feel particularly confident doing this change though; I've never touched the testing infrastructure and shell scripts aren't my primary area of expertise.

As an alternative, you could just merge your markup to master. :-)

Mantainers forbid direct PRs to master in boostorg/boost: https://github.com/boostorg/boost/pull/143 Joaquín M López Muñoz

Peter Dimov

1:03 p.m.

Joaquin M López Muñoz wrote:

...

Mantainers forbid direct PRs to master in boostorg/boost:

Does the repo-local markup not work on master?

Joaquin M López Muñoz

2:23 p.m.

El 26/10/2017 a las 15:03, Peter Dimov via Boost escribió:

...

Joaquin M López Muñoz wrote:

...
Mantainers forbid direct PRs to master in boostorg/boost:

Does the repo-local markup not work on master?

Wow, I didn't know this possibility even existed, and seems from the archives that the thing is pretty recent: https://lists.boost.org/Archives/boost/2017/10/238905.php It is no clear to me how this works: is this consolidating script something that Daniel runs locally every so often to manually update the superproject xml, or is the process done as an integrated step by reporting tools? I guess it's the former, since I couldn't see any relevant code change in boostorg/regression's commit history. Joaquín M López Muñoz

Daniel James

2:31 p.m.

On 26 October 2017 at 15:23, Joaquin M López Muñoz via Boost wrote:

...

El 26/10/2017 a las 15:03, Peter Dimov via Boost escribió:

...
Joaquin M López Muñoz wrote:

...
Mantainers forbid direct PRs to master in boostorg/boost:

You can for a change that's already on develop, or could just ask someone to cherry-pick it.

...

...
Does the repo-local markup not work on master?

Not for repos that aren't in master yet.

...

Wow, I didn't know this possibility even existed, and seems from the archives that the thing is pretty recent:

https://lists.boost.org/Archives/boost/2017/10/238905.php

It is no clear to me how this works: is this consolidating script something that Daniel runs locally every so often to manually update the superproject xml,

It's updated by a cron job once an hour, like the maintainer list.

Peter Dimov

3:04 p.m.

Daniel James wrote:

...

...
...
Does the repo-local markup not work on master?

Not for repos that aren't in master yet.

Doh. Looking at the differences between the develop and master explicit-failures-markup.xml, looks like most of the things not on master need to be there, but cherry-picking everything properly is a pain. Authors should be encouraged to move the markup to their repos. There are differences in - beast - context - contract (we know about this one) - coroutine - coroutine2 - fusion - hana - icl - poly_collection I think that any further PRs against the global status/explicit-failures-markup.xml need to be declined with the suggestion that the author moves the markup to the repo. (This doesn't at present help Lorenzo but if the only changes to cherry-pick are his things would be simpler.)

Lorenzo Caminiti

4:44 p.m.

On Thu, Oct 26, 2017 at 8:04 AM, Peter Dimov via Boost wrote:

...

(This doesn't at present help Lorenzo but if the only changes to cherry-pick are his things would be simpler.)

I will likely have more changes to make for Boost.Contract to explicit-failures-markup.xml before I can push the lib out of develop onto master (Boost.Contract has *lots* of tests which combined with the many regression toolsets make investigating and marking these failures quite a bit of work). I'm worried cherry-picking won't be an efficient solution for Boost.Contract because I will keep changing libs/contract/meta/explicit-failures-markup.xml on the develop branch often for another month or so... What's the best way for me to address this and get my markups to show on develop regression test report? Thanks a lot! --Lorenzo

Peter Dimov

4:56 p.m.

Lorenzo Caminiti wrote: ...

...

(Boost.Contract has *lots* of tests which combined with the many regression toolsets make investigating and marking these failures quite a bit of work).

This isn't markup-related, but given that your tests depend on C++11 lambdas, you probably need to consider adding a requirement for cxx11_lambdas in your test Jamfile, as explained here: http://www.boost.org/doc/libs/1_65_1/libs/config/doc/html/boost_config/build...

...

What's the best way for me to address this and get my markups to show on develop regression test report?

I've changed the report script per my earlier suggestions, we'll see how it goes. https://github.com/boostorg/regression/commit/b55fda7e5d28ad148dd9014438afa5...

Joaquin M López Muñoz

6:10 p.m.

El 26/10/2017 a las 18:56, Peter Dimov via Boost escribió:

...

Lorenzo Caminiti wrote:

...

...
(Boost.Contract has *lots* of tests which combined with the many regression toolsets make investigating and marking these failures quite a bit of work).

This isn't markup-related, but given that your tests depend on C++11 lambdas, you probably need to consider adding a requirement for cxx11_lambdas in your test Jamfile, as explained here:

[...]

I consider this superior to markup for disabling toolsets, but unfortunately resulting regression reports look ugly. You might consider taking a look at: https://lists.boost.org/Archives/boost/2017/10/239442.php Joaquín M López Muñoz

Peter Dimov

10:05 p.m.

...

I've changed the report script per my earlier suggestions, we'll see how it goes.

OK, looks like it worked (at the second try.) http://www.boost.org/development/tests/develop/developer/contract.html (Although I see that the page ends abruptly with "Error extracting file: Error code 2 - A generic error in the zipfile format was detected." - let's hope that this is an intermittent error.)

Lorenzo Caminiti

27 Oct 27 Oct

9:30 p.m.

On Thu, Oct 26, 2017 at 3:05 PM, Peter Dimov via Boost wrote:

...

...
I've changed the report script per my earlier suggestions, we'll see how it goes.

OK, looks like it worked (at the second try.)

http://www.boost.org/development/tests/develop/developer/contract.html

(Although I see that the page ends abruptly with "Error extracting file: Error code 2 - A generic error in the zipfile format was detected." - let's hope that this is an intermittent error.)

Yes, this works now. I can see the entire page with all tests. All my failure markups show correctly too. Thanks a lot for fixing this! --Lorenzo

Peter Dimov

26 Oct 26 Oct

2:42 p.m.

Joaquin M López Muñoz wrote:

...

El 26/10/2017 a las 15:03, Peter Dimov via Boost escribió:

...
Joaquin M López Muñoz wrote:

...
Mantainers forbid direct PRs to master in boostorg/boost:

Does the repo-local markup not work on master?

Wow, I didn't know this possibility even existed, and seems from the archives that the thing is pretty recent:

https://lists.boost.org/Archives/boost/2017/10/238905.php

It is no clear to me how this works: is this consolidating script something that Daniel runs locally every so often to manually update the superproject xml,

It's probably a cron job like the other boost-commitbot updates. https://github.com/boostorg/boost/commit/12f08779ed066dbb1e3c7dca725f86d22e4...

2587

Age (days ago)

2590

Last active (days ago)

List overview

Download

19 comments

5 participants

participants (5)

Daniel James
Edward Diener
Joaquin M López Muñoz
Lorenzo Caminiti
Peter Dimov