Best Practices for Surviving the Boost Test Gauntlet?

older
boost_check_library.py: correct...

Vinnie Falco

8 Aug 2017 8 Aug '17

2:12 p.m.

Beast won the first battle, but it looks like it is losing the war: http://www.boost.org/development/tests/develop/developer/beast.html I've never heard of many of these platforms let alone know how to go about fixing the compile, link, and runtime errors. It seems like deallocate() is not part of some standard library's allocator_traits? Link error on operator delete from Boost.System? A lot of little thing. Is there a document or resource which explains the various peculiarities and idiosyncrasies of these platforms and toolchains that I might have a means of fixing them without actually installing them locally (which is unlikely)? Thanks

Show replies by date

Raffi Enficiaud

9 Aug 9 Aug

9:15 p.m.

Le 08.08.17 à 16:12, Vinnie Falco via Boost a écrit :

...

Beast won the first battle, but it looks like it is losing the war:

http://www.boost.org/development/tests/develop/developer/beast.html

I've never heard of many of these platforms let alone know how to go about fixing the compile, link, and runtime errors. It seems like deallocate() is not part of some standard library's allocator_traits? Link error on operator delete from Boost.System? A lot of little thing.

Is there a document or resource which explains the various peculiarities and idiosyncrasies of these platforms and toolchains that I might have a means of fixing them without actually installing them locally (which is unlikely)?

Thanks

It seems that you redeveloped a testing framework, and this is causing some of the issues we can read there. I do not know how important is this internal framework to you, but you may go for another test framework and focus on the added value of your library instead. I am obviously biased toward boost.test :) Best, Raffi

Vinnie Falco

9:34 p.m.

On Wed, Aug 9, 2017 at 2:15 PM, Raffi Enficiaud via Boost wrote:

...

Le 08.08.17 à 16:12, Vinnie Falco via Boost a écrit : It seems that you redeveloped a testing framework, and this is causing some of the issues we can read there.

Is there a link to the evidence of this so that I can come up with a fix?

...

I do not know how important is this internal framework to you, but you may go for another test framework and focus on the added value of your library instead.

Changing test frameworks means refactoring all the tests - I'm sure you can understand why I might be reluctant to do that right now. Thanks

Edward Diener

10:01 p.m.

On 8/9/2017 5:34 PM, Vinnie Falco via Boost wrote:

...

On Wed, Aug 9, 2017 at 2:15 PM, Raffi Enficiaud via Boost wrote:

...
Le 08.08.17 à 16:12, Vinnie Falco via Boost a écrit : It seems that you redeveloped a testing framework, and this is causing some of the issues we can read there.

Is there a link to the evidence of this so that I can come up with a fix?

...
I do not know how important is this internal framework to you, but you may go for another test framework and focus on the added value of your library instead.

Changing test frameworks means refactoring all the tests - I'm sure you can understand why I might be reluctant to do that right now.

It is understandable but Boost comes with two testing frameworks built-in. There is Boost Test, which Raffi Enficiaud mentions and is pretty full-featured, and there is also Boost's lightweight_test.hpp, which I have found adequate for testing vmd and tti. It is always easier using what is already available, if it has the testing facilities you need, than to create your own.

...

Thanks

Raffi Enficiaud

10:11 p.m.

Le 09.08.17 à 23:34, Vinnie Falco via Boost a écrit :

...

On Wed, Aug 9, 2017 at 2:15 PM, Raffi Enficiaud via Boost wrote:

...
Le 08.08.17 à 16:12, Vinnie Falco via Boost a écrit : It seems that you redeveloped a testing framework, and this is causing some of the issues we can read there.

Is there a link to the evidence of this so that I can come up with a fix?

I was randomly clicking on a link, like this one: http://www.boost.org/development/tests/develop/developer/output/teeks99-09-m... Now, I do not know if this is relevant. For preventing running the tests on unsupported compilers, you may use b2 features, which will clear out some of the columns: http://www.boost.org/build/doc/html/bbv2/overview/targets.html#bbv2.overview... example in project: https://github.com/boostorg/test/blob/develop/test/Jamfile.v2#L30

...

...
I do not know how important is this internal framework to you, but you may go for another test framework and focus on the added value of your library instead.

Changing test frameworks means refactoring all the tests - I'm sure you can understand why I might be reluctant to do that right now.

When is now and when is later? If now means boost 1.65, then yes, this is not possible. If this is 1.66, I would say it is not such a big deal. OTOH, if later is never ... Your call anyway :) Raffi

Vinnie Falco

10:25 p.m.

On Wed, Aug 9, 2017 at 3:11 PM, Raffi Enficiaud via Boost wrote:

...

Le 09.08.17 à 23:34, Vinnie Falco via Boost a écrit : I was randomly clicking on a link, like this one:

http://www.boost.org/development/tests/develop/developer/output/teeks99-09-m...

This has nothing to do with the test framework, it just happens that suite.hpp is one of the first files included and therefore the first to generate an error. The problem in this case is that the toolchain is using Visual Studio 2012 which does not have sufficient support for C++11, and in particular insufficient support for "noexcept" as indicated here: https://msdn.microsoft.com/en-us/library/hh567368.aspx

...

For preventing running the tests on unsupported compilers, you may use b2 features, which will clear out some of the columns: http://www.boost.org/build/doc/html/bbv2/overview/targets.html#bbv2.overview...

Thanks, this is helpful!

...

When is now and when is later? If now means boost 1.65, then yes, this is not possible. If this is 1.66, I would say it is not such a big deal.

My schedule for 2017 is very tight. There is a backlog of Beast issues that I want to address for its debut in 1.66.0 as some of them may introduce API changes which I would rather do now instead of after putting out the first non-beta version. Boost.Asio is going to be updated at some random time in 2017 and I will have to quickly port Beast for those changes. And Beast needs to have some time without any changes prior to the freeze for 1.66.0, so that any lingering defects can be flushed out. There's a fair amount of test code and it makes heavy use of coroutines, threads, and helper classes (Beast's unit test framework is class-driven where each test is represented by a derived class). I have in mind to try out the lightweight framework after 1.66.0, which will be part of a larger push to bring the coverage as close to 100% as possible but I don't have the resources for it this year. So I am very much stuck with what I have. So far there is nothing in the build output to suggest that the testing framework is at fault but I will keep a close eye on it. What I'm looking for is the secret document that I am SURE is stashed away somewhere on a Google Drive account, or maybe a USB stick, which explains all of the common errors that happen in the Boost test environment and how they might be worked around. For example, that early versions of the standard library do not have full support for std::allocator_traits even though they claim to. I've since fixed that issue here: https://github.com/boostorg/beast/pull/725/commits/017510215122c013579f6c746... But as you know the turnaround time on cycling a library through the tests is quite long, so if I could get a copy of that document which has solutions to common problems ahead of time, then I could save several length iterations. Thanks

Gavin Lambert

11:04 p.m.

On 10/08/2017 10:25, Vinnie Falco wrote:

...

The problem in this case is that the toolchain is using Visual Studio 2012 which does not have sufficient support for C++11, and in particular insufficient support for "noexcept" as indicated here:

https://msdn.microsoft.com/en-us/library/hh567368.aspx

Are you using the BOOST_NOEXCEPT* macros? They resolve to nothing (or in some cases to throw()) on older compilers. In general for most compatibility issues there is a macro somewhere, either in Boost.Config or elsewhere.

Vinnie Falco

11:07 p.m.

On Wed, Aug 9, 2017 at 4:04 PM, Gavin Lambert via Boost wrote:

...

Are you using the BOOST_NOEXCEPT* macros? They resolve to nothing (or in some cases to throw()) on older compilers.

No, because Beast requires C++11.

...

In general for most compatibility issues there is a macro somewhere, either in Boost.Config or elsewhere.

I know, and its amazing the amount of work and dedication that some authors go through to keep their libraries working with old compilers. Unfortunately that level of support is not something I have resources to maintain in Beast. I can only support Beast on Visual Studio 2015 with Update 3 or later. Thanks

Niall Douglas

11:13 p.m.

...

I can only support Beast on Visual Studio 2015 with Update 3 or later.

The way I handled this in AFIO v1 was to add C++ feature requirements in Boost.Build sufficient to exclude any Visual Studio before some version X. I also strongly recommend you add #error trapping of ancient compilers in your config.hpp. Otherwise you'll be plagued in years to come by users complaining about unobvious error messages when they do weird things with some older version of your library. I personally would even go so far as to #error out on any compiler which is not MSVC, clang nor GCC, and versions before specific hardcoded minimum versions. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Vinnie Falco

11:14 p.m.

On Wed, Aug 9, 2017 at 4:13 PM, Niall Douglas via Boost wrote:

...

I also strongly recommend you add #error trapping of ancient compilers

Yes https://github.com/boostorg/beast/blob/1366302a29595b7300f8bd3481c459f8f4d0f...

Gavin Lambert

11:18 p.m.

On 10/08/2017 11:07, Vinnie Falco wrote:

...

On Wed, Aug 9, 2017 at 4:04 PM, Gavin Lambert wrote:

...
Are you using the BOOST_NOEXCEPT* macros? They resolve to nothing (or in some cases to throw()) on older compilers.

No, because Beast requires C++11.

...
In general for most compatibility issues there is a macro somewhere, either in Boost.Config or elsewhere.

I know, and its amazing the amount of work and dedication that some authors go through to keep their libraries working with old compilers. Unfortunately that level of support is not something I have resources to maintain in Beast.

I can only support Beast on Visual Studio 2015 with Update 3 or later.

Then it sounds like you should be putting something in the Jamfile that tells it not to attempt to build on older compilers that you intend to be incompatible with. (I don't know the specifics, though, but you could perhaps look at some of the other existing C++11/14-only libraries for inspiration.) Also, FYI, VS 12.0 is actually Visual Studio 2013.

Edward Diener

10 Aug 10 Aug

1:57 a.m.

On 8/9/2017 7:07 PM, Vinnie Falco via Boost wrote:

...

On Wed, Aug 9, 2017 at 4:04 PM, Gavin Lambert via Boost wrote:

...
Are you using the BOOST_NOEXCEPT* macros? They resolve to nothing (or in some cases to throw()) on older compilers.

No, because Beast requires C++11.

...
In general for most compatibility issues there is a macro somewhere, either in Boost.Config or elsewhere.

I know, and its amazing the amount of work and dedication that some authors go through to keep their libraries working with old compilers. Unfortunately that level of support is not something I have resources to maintain in Beast.

This is almost exclusively for libraries which have been around for a long time. Boost fully accepts the fact that your library may need a particular level of c++ compliance. The important thing is to document the level of c++ compliance needed by your library. Once you have done that then it is the end-user's problem if they are using a compiler that does not meet the level of c++ that your library needs if they wish to use your library.

...

I can only support Beast on Visual Studio 2015 with Update 3 or later.

Thanks

Gavin Lambert

10:56 p.m.

On 10/08/2017 13:57, Edward Diener wrote:

...

This is almost exclusively for libraries which have been around for a long time. Boost fully accepts the fact that your library may need a particular level of c++ compliance. The important thing is to document the level of c++ compliance needed by your library. Once you have done that then it is the end-user's problem if they are using a compiler that does not meet the level of c++ that your library needs if they wish to use your library.

While that's true, it's also polite to have your Jamfile configured to skip compiling on the unsupported platforms, so that the user doesn't have to explicitly opt out with --without-xxx. Although it's not as big a deal for header-only libraries, since the default build doesn't build the tests anyway, IIRC.

Edward Diener

11 Aug 11 Aug

12:12 a.m.

On 8/10/2017 6:56 PM, Gavin Lambert via Boost wrote:

...

On 10/08/2017 13:57, Edward Diener wrote:

...
This is almost exclusively for libraries which have been around for a long time. Boost fully accepts the fact that your library may need a particular level of c++ compliance. The important thing is to document the level of c++ compliance needed by your library. Once you have done that then it is the end-user's problem if they are using a compiler that does not meet the level of c++ that your library needs if they wish to use your library.

While that's true, it's also polite to have your Jamfile configured to skip compiling on the unsupported platforms, so that the user doesn't have to explicitly opt out with --without-xxx.

I agree with you.

...

Although it's not as big a deal for header-only libraries, since the default build doesn't build the tests anyway, IIRC.

Actually you can do this for tests also, but I agree that it is more important to do this for library builds.

Robert Ramey

9 Aug 9 Aug

11:34 p.m.

On 8/8/17 7:12 AM, Vinnie Falco via Boost wrote:

...

Beast won the first battle, but it looks like it is losing the war:

http://www.boost.org/development/tests/develop/developer/beast.html

I've never heard of many of these platforms let alone know how to go about fixing the compile, link, and runtime errors. It seems like deallocate() is not part of some standard library's allocator_traits? Link error on operator delete from Boost.System? A lot of little thing.

Is there a document or resource which explains the various peculiarities and idiosyncrasies of these platforms and toolchains that I might have a means of fixing them without actually installing them locally (which is unlikely)?

I looked at your results, and they don't look half bad for a first attempt. I can share you the benefit of years of experience dealing with this. It's much better now that it used to be. a) Make sure your code aims for standard C++. b) Any failures are now due to compilers/libraries failing to meet the standard. c) Now study documentation on config.hpp. It's very complete, readable and not at all difficult to use. It basically entails tweaking your code as suggested by the documentation. You'll find that a lot of problems disappear after a relatively short session as many problems are in a header show up in every test. d) You can tweak your test/jamfile so that it skips tests which are not applicable. For example if your library doesn't have header X you can arrange to skip those tests which require header X. The information is in the documentation for config.hpp. Also the test/bjam file for the serialization library does this and you see the syntax there. e) If you follow the above to the end, you will find that your tests will run on all versions of C++ since you've excluded those tests/compiler combinations which cannot pass. f) FWIW I use my own variation of lightweight test. As long as your test returns a 1 (failure) or 0 (success) I don't think it matters which test framework you use. g) When you get all done with the above, you'll have most of the tests running on most compilers. The failures are just that - things that can't be made to pass without mucking up your library. Just note that it's the vendors problem and move on. h) When all this is done, get down on your knees say a prayer of thanks to the almighty that your library is header only. Robert Ramey

2664

Age (days ago)

2667

Last active (days ago)

List overview

Download

14 comments

6 participants

participants (6)

Edward Diener
Gavin Lambert
Niall Douglas
Raffi Enficiaud
Robert Ramey
Vinnie Falco