[1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

newer
How to make boost write_xml not...

John Maddock

4 Nov 2019 4 Nov '19

6:59 p.m.

This change: https://github.com/boostorg/test/commit/1c4533f7654d4c049cc6a34a43a5c10cde5b... To Boost test (which I'm sure has been long signaled, I just wasn't aware of it), breaks one Math lib test (develop and master), and quite a number of Fibre tests (also develop and master), the Math failure should be fixed in develop (https://github.com/boostorg/math/commit/1d04bf714e809d386088ce135e3f9ea04caa...) and will get merged to master once CI is green. Not sure if anything is in the works for Fiber. Just a wee heads-up. Best, John. -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Show replies by date

Oliver Kowalke

4 Nov 4 Nov

9:12 p.m.

Am Mo., 4. Nov. 2019 um 19:59 Uhr schrieb John Maddock via Boost :

...

Not sure if anything is in the works for Fiber.

wasn't aware of the changes, so no fixes are in the pipeline for boost.fiber

Raffi Enficiaud

10:56 p.m.

On 04.11.19 19:59, John Maddock via Boost wrote:

...

This change: https://github.com/boostorg/test/commit/1c4533f7654d4c049cc6a34a43a5c10cde5b...

Oh, sorry about that. Reverting the lines right now.

...

To Boost test (which I'm sure has been long signaled, I just wasn't aware of it), breaks one Math lib test (develop and master), and quite a number of Fibre tests (also develop and master), the Math failure should be fixed in develop (https://github.com/boostorg/math/commit/1d04bf714e809d386088ce135e3f9ea04caa...) and will get merged to master once CI is green. Not sure if anything is in the works for Fiber.

Just a wee heads-up.

Best, John.

pbristow＠hetp.u-net.com

5 Nov 5 Nov

12:41 p.m.

...

-----Original Message----- From: Boost On Behalf Of Raffi Enficiaud via Boost Sent: 4 November 2019 22:57 To: boost@lists.boost.org Cc: Raffi Enficiaud Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

On 04.11.19 19:59, John Maddock via Boost wrote:

...
This change: https://github.com/boostorg/test/commit/1c4533f7654d4c049cc6a34a43a5c1 0cde5bdd8b#diff-efad11c1b810a644deec92a42abe330e

Oh, sorry about that. Reverting the lines right now.

Actually I like the change - did we need 'framework'? Anyone using this is perhaps relying on a detail? (The need for it in the Math test case was to ensure that the BOOST_TEST_MESSAGE("my_message") were actually visible. I think it would be better if messages were visible by default, not something one needs to ask for specially. I also was unconvinced the setting the environment variable BOOST_TEST_MESSAGE worked but I'll check this out and raise a bug report.) But definitely a note in the release notes would be good. The 'fix' is to rename unit_test_framework:: to unit_test:: Paul

...

...
To Boost test (which I'm sure has been long signaled, I just wasn't aware of it), breaks one Math lib test (develop and master), and quite a number of Fibre tests (also develop and master), the Math failure should be fixed in develop (https://github.com/boostorg/math/commit/1d04bf714e809d386088ce135e3f9 ea04caac2b8) and will get merged to master once CI is green. Not sure if anything is in the works for Fiber.

Just a wee heads-up.

Best, John.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

pbristow＠hetp.u-net.com

4:03 p.m.

...

-----Original Message----- From: Boost On Behalf Of Paul A Bristow via Boost Sent: 5 November 2019 12:42 To: boost@lists.boost.org Cc: pbristow@hetp.u-net.com Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

...
-----Original Message----- From: Boost On Behalf Of Raffi Enficiaud via Boost Sent: 4 November 2019 22:57 To: boost@lists.boost.org Cc: Raffi Enficiaud Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

On 04.11.19 19:59, John Maddock via Boost wrote:

...
This change: https://github.com/boostorg/test/commit/1c4533f7654d4c049cc6a34a43a5 c1 0cde5bdd8b#diff-efad11c1b810a644deec92a42abe330e

Oh, sorry about that. Reverting the lines right now.

Actually I like the change - did we need 'framework'?

Anyone using this is perhaps relying on a detail?

(The need for it in the Math test case was to ensure that the BOOST_TEST_MESSAGE("my_message") were actually visible. I think it would be better if messages were visible by default, not something one needs to ask for specially. I also was unconvinced the setting the environment variable BOOST_TEST_MESSAGE worked but I'll check this out and raise a bug report.)

I've finally found this exchange explaining the behaviour. http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAGE-td46... 8.html I think it is very counter-intuitive that messages only emerge to the normal output if there is an error or --log_level="message" And that #define BOOST_TEST_LOG_LEVEL "message" does not appear to have the same effect as --log_level="message" which is also surprising, and in my book a buglet? I suggest that this behaviour should be changed so that BOOST_TEST_MESSAGE *always* produces output. BOOST_AUTO_TEST_CASE(test_message) { BOOST_TEST_MESSAGE("\nTest BOOST_TEST_MESSAGE.\n"); } // BOOST_AUTO_TEST_CASE(test_message) boost_test_message_test.vcxproj -> I:\Cpp\math\x64\Release\boost_test_message_test.exe should output Running 1 test case... Test BOOST_TEST_MESSAGE. *** No errors detected Otherwise I am sure I will not be the only person confused. At the very least the documentation should highlight that ones expectation may not match what actually happens. Paul

Raffi Enficiaud

6:48 p.m.

On 05.11.19 17:03, Paul A Bristow via Boost wrote:

...

...
[snip]

Actually I like the change - did we need 'framework'?

"#me too" but this is not the right time to introduce any breaking changes, especially without notice. It is used nowhere in boost.test but I have not checked within boost (and certainly not outside) before making that change (that was also unrelated to the issue being solved). Some legacy code, a proper deprecation should be emitted, at least in the change logs. I do not know if namespaces can have deprecation attributes, that would be handy.

...

...
Anyone using this is perhaps relying on a detail?

(The need for it in the Math test case was to ensure that the BOOST_TEST_MESSAGE("my_message") were actually visible. I think it would be better if messages were visible by default, not something one needs to ask for specially. I also was unconvinced the setting the environment variable BOOST_TEST_MESSAGE worked but I'll check this out and raise a bug report.)

I've finally found this exchange explaining the behaviour.

http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAGE-td46... 8.html

I think it is very counter-intuitive that messages only emerge to the normal output if there is an error or --log_level="message"

And that

#define BOOST_TEST_LOG_LEVEL "message"

does not appear to have the same effect as --log_level="message"

The current documentation never mentions the usage through a macro as you are using in the snippet above. Please see below about the relevant sections.

...

which is also surprising, and in my book a buglet?

I suggest that this behaviour should be changed so that BOOST_TEST_MESSAGE *always* produces output.

BOOST_AUTO_TEST_CASE(test_message) { BOOST_TEST_MESSAGE("\nTest BOOST_TEST_MESSAGE.\n"); } // BOOST_AUTO_TEST_CASE(test_message)

boost_test_message_test.vcxproj -> I:\Cpp\math\x64\Release\boost_test_message_test.exe

should output

Running 1 test case...

Test BOOST_TEST_MESSAGE.

*** No errors detected

I tend to disagree with that. What I am expecting from my tests is the report of "true positives", ie. assertions that fails for good reasons like regressions. All the rest I consider as information that distracts, or information printed only for non-regular/normal use like debugging. I would for instance put a message or a warning that I can easily activate if something goes wrong and needs my attention, until I fix the issue. Then I would go back to the normal mode. This is also the approach taken by tools like ctest or b2: normal logging level prints only a very high level status, verbose logs are redirected to file for potential inspection, details are printed on error. The behaviour of Boost.Test in that regard is documented in various places, mostly in the runtime arguments (command line) and in the logging part: * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_out... * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_out... * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_out... (even with its own section) * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_refe... (indicates the environment variable as well) * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/runtime_... although the section about environment variables can be clarified

...

Otherwise I am sure I will not be the only person confused.

At the very least the documentation should highlight that ones expectation may not match what actually happens.

I have to say that the current doc is sometimes more than complete with some redundant parts and confusing information. If you think of a better way to organize it or a better wording ("me not English native"), please let me know, I would be happy to improve it. Thanks! Raffi PS: and sorry about breaking your code again so close to the beta.

pbristow＠hetp.u-net.com

6 Nov 6 Nov

9:49 a.m.

...

-----Original Message----- From: Boost On Behalf Of Raffi Enficiaud via Boost Sent: 5 November 2019 18:49 To: boost@lists.boost.org Cc: Raffi Enficiaud Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

On 05.11.19 17:03, Paul A Bristow via Boost wrote:

...
...
[snip]

Actually I like the change - did we need 'framework'?

"#me too" but this is not the right time to introduce any breaking changes, especially without notice. It is used nowhere in boost.test but I have not checked within boost (and certainly not outside) before making that change (that was also unrelated to the issue being solved). Some legacy code, a proper deprecation should be emitted, at least in the change logs.

I do not know if namespaces can have deprecation attributes, that would be handy.

...
...
Anyone using this is perhaps relying on a detail?

(The need for it in the Math test case was to ensure that the BOOST_TEST_MESSAGE("my_message") were actually visible. I think it would be better if messages were visible by default, not something one needs to ask for specially. I also was unconvinced the setting the environment variable BOOST_TEST_MESSAGE worked but I'll check this out and raise a bug report.)

I've finally found this exchange explaining the behaviour.

http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAG E-td467539 8.html

I think it is very counter-intuitive that messages only emerge to the normal output if there is an error or --log_level="message"

And that

#define BOOST_TEST_LOG_LEVEL "message"

does not appear to have the same effect as --log_level="message"

The current documentation never mentions the usage through a macro as you are using in the snippet above. Please see below about the relevant sections.

...
which is also surprising, and in my book a buglet?

I suggest that this behaviour should be changed so that BOOST_TEST_MESSAGE *always* produces output.

BOOST_AUTO_TEST_CASE(test_message) { BOOST_TEST_MESSAGE("\nTest BOOST_TEST_MESSAGE.\n"); } // BOOST_AUTO_TEST_CASE(test_message)

boost_test_message_test.vcxproj -> I:\Cpp\math\x64\Release\boost_test_message_test.exe

should output

Running 1 test case...

Test BOOST_TEST_MESSAGE.

*** No errors detected

I tend to disagree with that. What I am expecting from my tests is the report of "true positives", ie. assertions that fails for good reasons like regressions. All the rest I consider as information that distracts, or information printed only for non- regular/normal use like debugging. I would for instance put a message or a warning that I can easily activate if something goes wrong and needs my attention, until I fix the issue. Then I would go back to the normal mode.

This is also the approach taken by tools like ctest or b2: normal logging level prints only a very high level status, verbose logs are redirected to file for

There are times when eminently sensible suggestions (like yours and mine) are not the best thing to do. I agree that the benefit isn't worth causing downstream trouble. But I would still like some warning in the documentation. I've written notes-to-self: // #define BOOST_TEST_LOG_LEVEL "all" or /DBOOST_TEST_LOG_LEVEL // Do not output unless there is an error! // http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAGE-td46... 8.html // run with --log_level="message" always gives message. I use a VS IDE Custom build step "$(TargetDir)$(TargetName).exe" --log_level="message" for testing console apps. Paul potential

...

inspection, details are printed on error.

The behaviour of Boost.Test in that regard is documented in various places, mostly in the runtime arguments (command line) and in the logging part:

* https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_out... ut/log_formats.html * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_out... ut/log_formats/test_log_output.html * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_out... ut/test_tools_support_for_logging/test_output_macro_message.html (even with its own section) * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_refe... nce/rt_param_reference/log_level.html (indicates the environment variable as well) * https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/runtime_... onfig.html although the section about environment variables can be clarified

...
Otherwise I am sure I will not be the only person confused.

At the very least the documentation should highlight that ones expectation may not match what actually happens.

I have to say that the current doc is sometimes more than complete with some redundant parts and confusing information. If you think of a better way to organize it or a better wording ("me not English native"), please let me know, I would be happy to improve it.

Thanks! Raffi

PS: and sorry about breaking your code again so close to the beta.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Raffi Enficiaud

7 Nov 7 Nov

6:47 a.m.

On 06.11.19 10:49, Paul A Bristow via Boost wrote:

...

There are times when eminently sensible suggestions (like yours and mine) are not the best thing to do.

I agree that the benefit isn't worth causing downstream trouble.

But I would still like some warning in the documentation. I've written notes-to-self:

I am all for helping you there. My question is: where should I put this warning? It is not clear to me if you expect this warning in the doc, and if so, which section of the doc.

pbristow＠hetp.u-net.com

9:51 a.m.

...

-----Original Message----- From: Boost On Behalf Of Raffi Enficiaud via Boost Sent: 7 November 2019 06:47 To: boost@lists.boost.org Cc: Raffi Enficiaud Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

On 06.11.19 10:49, Paul A Bristow via Boost wrote:

...
There are times when eminently sensible suggestions (like yours and mine) are not the best thing to do.

I agree that the benefit isn't worth causing downstream trouble.

But I would still like some warning in the documentation. I've written notes-to-self:

I am all for helping you there. My question is: where should I put this warning? It is not clear to me if you expect this warning in the doc, and if so, which section of the doc.

https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_refe... Doesn't make clear to me what the default level is 'out of the box'? And perhaps warn on other page that it isn't output at the default log level. https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_refe... should also suggest that running with --log_level="message2 is required to get this to actually output anything. https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_out... and this too. (I am ashamed to report that I did not follow my own injunction to use hyperlinks to get to this page ☹ As they say, "you can lead a horse to water, but you can't make him drink"). I also didn't find that #define BOOST_TEST_LOG_LEVEL message ort "message" had the desired effect. Many people might expect this (and consider it desirable to force the messages out. Thanks HTH Paul

1846

Age (days ago)

1849

Last active (days ago)

List overview

Download

8 comments

4 participants

participants (4)

John Maddock
Oliver Kowalke
pbristow＠hetp.u-net.com
Raffi Enficiaud