On 7/26/2017 3:33 PM, Robert Ramey via Boost wrote:

On 7/26/17 11:49 AM, Edward Diener via Boost wrote:

...

Following John Maddock's appeal for practical solutions related to the move to CMake, I would like to know what the CMake equivalent is to the Boost Build unit test functionality.

I can tell you what I did for the serialization library. This predates the current discussions so will likely be considered out of date by the current contributors to the discussion. On the other hand, it does actually work and you can test it yourself.

Robert I appreciate your answer. Is this the CMakeLists.txt in the CMake subdirectory of the serialization library ? Remember I am talking about header-only libraries where nothing is "built" and I am only trying to run the unit tests in the library's 'test' subdirectory using CMake rather than Boost Build using CMake's equivalent of a script file rather than a Boost Build jamfile.

...

In other words what do I write for CMake in order to do a Boost Build compile, compile-fail, link, link-fail, run, and run-fail unit tests ?

One adds targets (which for me is just the name of the source/excutable without the cpp, exe, etc. to a list to get compile and run functionality. Its pretty simple.

I want to see actual cmake code. Saying "Its pretty simple" only means it is pretty simple for those who already know CMake well. <RANT> I find the CMake documentation at https://cmake.org/cmake/help/v3.9/ nearly impossible to understand due to the fact that there is no overriding explanation of how CMake is supposed to work as practical examples of setting up CMake scripts. While I realize that all the information is supposedly there I have just about no idea how it all comes together. I am guessing that the only way I can really learn CMake is buying the "Mastering CMake" book, which I don't mind doing so if I have to do it. If I am wrong and someone would like to point me to an actual overview which explains CMake in a regular and meaningful way, rather than a bunch of loosely related topics which is the actual CMake documentation in the above URL, it would be appreciated. </RANT>

I couldn't make the compile-fail and run-fail targets to work so I just removed them from the cmake test suite. So if your thinking that CMake has a ways to go to be able to offer what we expect, you're right.

So you are saying that you cannot run a unit test with CMake that is meant to show a failure. Wow, what a brilliant system !

...

Does CMake have an equivalent to the Boost Build alias rule ? or the Boost Build project rule ?

LOL - I honestly don't remember what these rules do in boost build. But then I've only been using boost build for 15 years.

The project rule for unit tests is normally used to set usage requirements for all tests being run. The alias rule for unit tests is normally used as a means of grouping some unit tests together so that the end-user of the tests can test individual groups rather than all the tests.

...

The great majority of the Boost libraries are header-only libraries where Boost Build is used to run the unit tests and,

true and this case is pretty simple in both cmake and boost build.

Once again saying it is pretty simple in cmake isn't showing me anything.

...

optionally, Boost Build is used to build the documentation using a quickbook, boostbook, doxygen toolset to generate html and pdf files.

Personally, I could never make this tool chain work despite a large investment of effort. I gave up on doxygen and quickbook after concluding them to be essentially non-functional. I did manage to use the xml tools loaded with boost tools along with some downloaded FOP, a version of XMLMind to create toolchain which is about 100 times better and 1/100 as complex as the boost one for creating documents. So you're life will be better when you're forced to leave these behind.

I heartily disagree. Using quickbook and doxygen is a no-brainer for me, but there is no point of debating this. Are both perfect ? No. But using them makes creating documentation supremely easy. I did say that there is no Boost requirement for creating documentation using Boost Build. Some documentation is straight html. But for those libraries that do go the quickbook and doxygen way using Boost Build we need a CMake way of doing this if that is what the Boost Steering Committee is mandating.

While I realize that

...

building a Boost library, and making that built library available to end-users via CMake, has been the focus of most of the discussion so far regarding CMake and Boost,

You are correct. There are two independent issues related to this.

a) Replacing boost build with cmake b) providing CMake files to be helpful to users who use CMake to build their applications.

Boost should first address b) first while leaving a) untouched for now. If it can't manage to do that, then there is not point in trying to address b)

I am assuming the move to CMake also means that running unit tests

I think that is the goal

and building docs where applicable must also be done through CMake.

I don't see this as a requirement nor do I see it as necessary or desirable.

...

If I am wrong about this then I personally will be only too happy to continue using Boost Build to do these things.

LOL - it certainly looks that way.

I wouldn't panic yet. Nothing is actually going to happen until someone actually does something (besides offer advice). It doesn't seem that that is going to happen very soon, so you can just relax.

OK, I will relax and await the outcome of those, whoever they are, who are evidently taking on the work of providing CMake scripts for what is currently Boost Build tasks.

But you might want to experiment a little with the following useful orthogonal ideas.

a) investigate how CMake Find<your library> should work and consider adding the files necessary to support it to your library. This wouldn't mess up any of your other work.

b) Consider making a simple CMake project to build library and run tests such as I did for the serialization library. CMake gurus it seems dismiss this effort. But it has worked for me. The main benefit for me is that permits to to build a working IDE which I can used to build and debug. For me, this has turned out to be a great time saver.

I do not know how you learned enough of CMake to do what you want but, apropos my RANT, I am finding my ability to do this severely hampered by the piecemeal documentation which CMake deems its own docs.

Also I use boost build for running unit tests on my local system and it works very well for that. It's what I really depend on. It doesn't consume any effort to maintain (it was a MAJOR effort to setup). And I created my own program to display test results (library_test) so I'm very happy with this. One thing that CMake has that Boost Build doesn't have is CDash does have is the ability to upload test results to a special site where all test results from all users can be browsed. This is desperately needed for Boost but no one has made the effort to build it.

So all in all, I'm hoping that we can evolve to something better and more widely useful. But I don't see anyone actually undertaking efforts to make that happen. So I don't think you and I really have to worry about this all that much. We'll Boost build will likely out live you and me.

Robert Ramey

On 7/26/17 1:36 PM, Roger Leigh via Boost wrote:

are one way to generate doxygen docs. I'm sure something similar to this could be made to meet Boost's needs.

I've avoided addressing doxygen ... until you brought it up. This tool is probably the single biggest reason why software development documentation is soo very bad. Things would be much, much better it had never been invented. On the bright side, it does generate crappy documentation in the shortest time. Since authors are going to create crappy documentation regardless, I guess it's a win. Robert Ramey

On 27/07/2017 09:08, Vinnie Falco wrote:

On Wed, Jul 26, 2017 at 2:06 PM, Robert Ramey wrote:

...

I've avoided addressing doxygen ... until you brought it up. This tool is probably the single biggest reason why software development documentation is soo very bad. Things would be much, much better it had never been invented. On the bright side, it does generate crappy documentation in the shortest time. Since authors are going to create crappy documentation regardless, I guess it's a win.

I disagree. This page is generated from Doxygen and it looks great: http://vinniefalco.github.io/beast/beast/ref/beast__http__async_read/overloa...

One of the problems I've found with Doxygen is that many people don't actually get to that page, but rather only get as far as http://vinniefalco.github.io/beast/beast/ref/beast__http__async_read.html. Once you've encountered enough Doxygen docs you might eventually notice the little tiny "more" links and achieve enlightenment.

On 7/26/2017 4:36 PM, Roger Leigh via Boost wrote:

On 26/07/17 21:28, Edward Diener via Boost wrote:

...

On 7/26/2017 3:33 PM, Robert Ramey via Boost wrote:

...

On 7/26/17 11:49 AM, Edward Diener via Boost wrote:

...

optionally, Boost Build is used to build the documentation using a quickbook, boostbook, doxygen toolset to generate html and pdf files.

Personally, I could never make this tool chain work despite a large investment of effort. I gave up on doxygen and quickbook after concluding them to be essentially non-functional. I did manage to use the xml tools loaded with boost tools along with some downloaded FOP, a version of XMLMind to create toolchain which is about 100 times better and 1/100 as complex as the boost one for creating documents. So you're life will be better when you're forced to leave these behind.

I heartily disagree. Using quickbook and doxygen is a no-brainer for me, but there is no point of debating this. Are both perfect ? No. But using them makes creating documentation supremely easy.

I did say that there is no Boost requirement for creating documentation using Boost Build. Some documentation is straight html. But for those libraries that do go the quickbook and doxygen way using Boost Build we need a CMake way of doing this if that is what the Boost Steering Committee is mandating.

https://github.com/ome/ome-files-cpp/blob/master/docs/doxygen/CMakeLists.txt

https://github.com/ome/ome-files-cpp/blob/master/cmake/DoxygenCheck.cmake

are one way to generate doxygen docs. I'm sure something similar to this could be made to meet Boost's needs.

I certainly hope so since the creation of docs using the quickbook/boostbook/doxygen toolsets via a Boost Build jam file is pretty flexible and easy, once you understand the jam file structure for it.

Regards, Roger

...

I couldn't make the compile-fail and run-fail targets to work so I just removed them from the cmake test suite. So if your thinking that CMake has a ways to go to be able to offer what we expect, you're right.

So you are saying that you cannot run a unit test with CMake that is meant to show a failure. Wow, what a brilliant system !

Actually cmake's support for fail tests is richer than Boost.Build's. Here is cmake to implement simple compile fail testing where a regex on the error messages from the compiler must be matched for the test to pass: https://github.com/ned14/quickcpplib/blob/master/cmakelib/QuickCppLibMakeSta... This cmake reads the regex from the first line of the fail test's source code. There is lots of flexibility in cmake's fail test design. You can make tests which must compile but must fail to link. You can make tests which must complete within a certain time period. Lots of flexibility. ctest generates XML of all the test results. That can be merged with JUnit XML from the individual tests easily enough. And the whole lot sent up to CDash e.g. http://my.cdash.org/index.php?project=Boost.Outcome&date=2017-07-23 where fails and successes can be drilled into, and the cause figured out. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

...

Actually cmake's support for fail tests is richer than Boost.Build's.

Here is cmake to implement simple compile fail testing where a regex on the error messages from the compiler must be matched for the test to pass:

https://github.com/ned14/quickcpplib/blob/master/cmakelib/QuickCppLibMakeSta...

Thanks ! I will look at it. It does seem like a great deal of code just to run a bunch of tests.

Only four lines or so are needed to make a compile-fail test in cmake. You can extract those out from my code easily enough. The reason for all the code is that QuickCppLib automates everything. You simply point it at a directory tree laid out in boost format, fill in a few parameters like <library name>, and whoosh it all just works with a complete package of support for docs, sanitiser tests, C++ Modules, precompiled headers, versioning, mangling, the whole kaboosh. But that's cmake innovation. And I wouldn't recommend that for Boost. You guys need something as simple and as minimal as possible which matches, as close as possible, Boost.Build. Otherwise you'll never finish converting over the build system, you'll get 60% there and that long tail will become insurmountable given it's volunteer time. We saw exactly the same happen with the git migration, it's that long tail which kills you by causing people to quit the whole project rather than put up with the never ending criticism. I personally think that a build directory per configuration design is too dissimilar to Boost.Build that such a conversion could ever be finished without injecting paid death march hours. But I won't be doing the work, so it's up to those who are to choose their design poison.

...

This cmake reads the regex from the first line of the fail test's source code.

There is lots of flexibility in cmake's fail test design. You can make tests which must compile but must fail to link. You can make tests which must complete within a certain time period. Lots of flexibility.

ctest generates XML of all the test results. That can be merged with JUnit XML from the individual tests easily enough. And the whole lot sent up to CDash e.g. http://my.cdash.org/index.php?project=Boost.Outcome&date=2017-07-23 where fails and successes can be drilled into, and the cause figured out.

I have a feeling that CTest is not usable for the current Boost testing system, but maybe I am wrong.

CTest I think is more than able. The sole thing I've ever found it falls badly down on is running tests on real remote devices e.g. Android ones. It really wants to run on-device, not remotely. Still, it can be beaten into submission using evil trickery. CDash probably isn't up to Boost's requirements. But the XML CTest outputs is standardised, it's not even Kitware's XML design. So the tooling currently producing the Boost regression matrix etc ought to be very easy to teach to understand CTest XML output, and thus all becomes well. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On Wed, 2017-07-26 at 21:48 +0100, Roger Leigh via Boost wrote:

On 26/07/17 21:28, Edward Diener via Boost wrote:

...

On 7/26/2017 3:33 PM, Robert Ramey via Boost wrote:

...

On 7/26/17 11:49 AM, Edward Diener via Boost wrote:

...

I couldn't make the compile-fail and run-fail targets to work so I  just removed them from the cmake test suite.  So if your thinking that  CMake has a ways to go to be able to offer what we expect, you're right.

So you are saying that you cannot run a unit test with CMake that is  meant to show a failure. Wow, what a brilliant system !

ctest itself treats a nonzero return as failure.  It's pretty trivial to  run a wrapper to invert the status for expected failures, and customise  the test running in any way you see fit.  See the EXPECT_FAIL lines in

https://github.com/apache/xerces-c/blob/f7654ee16b1dc2fb6e2b68777895c5384cd5 1c28/tests/CMakeLists.txt#L176

and the wrapper logic in

https://github.com/apache/xerces-c/blob/f7654ee16b1dc2fb6e2b68777895c5384cd5 1c28/cmake/RunTest.cmake#L45

as one way to do this.

There's really no need for that because you can just set the test property WILL_FAIL to true, like I showed in my previous email.

On 7/26/2017 4:48 PM, Roger Leigh via Boost wrote:

On 26/07/17 21:28, Edward Diener via Boost wrote:

...

On 7/26/2017 3:33 PM, Robert Ramey via Boost wrote:

...

On 7/26/17 11:49 AM, Edward Diener via Boost wrote:

...

...

I couldn't make the compile-fail and run-fail targets to work so I just removed them from the cmake test suite. So if your thinking that CMake has a ways to go to be able to offer what we expect, you're right.

So you are saying that you cannot run a unit test with CMake that is meant to show a failure. Wow, what a brilliant system !

ctest itself treats a nonzero return as failure. It's pretty trivial to run a wrapper to invert the status for expected failures, and customise the test running in any way you see fit. See the EXPECT_FAIL lines in

https://github.com/apache/xerces-c/blob/f7654ee16b1dc2fb6e2b68777895c5384cd5...

and the wrapper logic in

https://github.com/apache/xerces-c/blob/f7654ee16b1dc2fb6e2b68777895c5384cd5...

as one way to do this.

That is just applicable to a run test. Current Boost Build also has compile and link tests, although I will admit I have never found uses for the link test. But in the compile test, only whether a compile is successful or not is measured, while the run test does follow CTest treating a nonzero return as failure. But is CTest really applicable as a CMake alternative to Boost testing as it now exists under Boost Build ? The documentation for CTest does not explain what it actually does amidst it enumeration of a trillion or so command line options, but why should it do that, right ?

Regards, Roger

On 7/26/2017 5:03 PM, Robert Ramey via Boost wrote:

On 7/26/17 1:28 PM, Edward Diener via Boost wrote:

...

On 7/26/2017 3:33 PM, Robert Ramey via Boost wrote:

...

On 7/26/17 11:49 AM, Edward Diener via Boost wrote:

...

Following John Maddock's appeal for practical solutions related to the move to CMake, I would like to know what the CMake equivalent is to the Boost Build unit test functionality.

I can tell you what I did for the serialization library. This predates the current discussions so will likely be considered out of date by the current contributors to the discussion. On the other hand, it does actually work and you can test it yourself.

Robert I appreciate your answer. Is this the CMakeLists.txt in the CMake subdirectory of the serialization library

right

? Remember I am talking about

...

header-only libraries where nothing is "built" and I am only trying to run the unit tests in the library's 'test' subdirectory using CMake rather than Boost Build using CMake's equivalent of a script file rather than a Boost Build jamfile.

right that's what it does. It builds the library and executes the tests

But the case I brought up is that there is no "build' of the library, since it is header only, but the library needs to execute its tests using CMake. That appears to me to be different from your case.

...

...

...

In other words what do I write for CMake in order to do a Boost Build compile, compile-fail, link, link-fail, run, and run-fail unit tests ?

One adds targets (which for me is just the name of the source/excutable without the cpp, exe, etc. to a list to get compile and run functionality. Its pretty simple.

I want to see actual cmake code. Saying "Its pretty simple" only means it is pretty simple for those who already know CMake well.

aaa - the CMakeLists.txt IS the "cmake code". You invoke CMake - I use the GUI version and it produces and makefile, or an IDE file (I generate the one for XCode). They you build the library (if it has one) and run the tests from the project it created for you. It's a two step process.

So you are saying that even when their is no actual library to be built, as is the case with a header only library, you still specify a CMakeLists.txt file as if you were building a library ?

...

<RANT>

I find the CMake documentation at https://cmake.org/cmake/help/v3.9/ nearly impossible to understand due to the fact that there is no overriding explanation of how CMake is supposed to work as practical examples of setting up CMake scripts. While I realize that all the information is supposedly there I have just about no idea how it all comes together. I am guessing that the only way I can really learn CMake is buying the "Mastering CMake" book, which I don't mind doing so if I have to do it. If I am wrong and someone would like to point me to an actual overview which explains CMake in a regular and meaningful way, rather than a bunch of loosely related topics which is the actual CMake documentation in the above URL, it would be appreciated.

This is a serious short coming of CMake. Given this it's a mystery that CMake has been successful. But then then look at all the competition it's ever worse!!! To addressed this, I wrote a walk through in the boost library inclubator to help library developers get started with CMake. It uses on the safe numerics library CMake as an example. This example/narrative addresses only header-only files.

Would you please give a URL ?

...

</RANT>

...

I couldn't make the compile-fail and run-fail targets to work so I just removed them from the cmake test suite. So if your thinking that CMake has a ways to go to be able to offer what we expect, you're right.

So you are saying that you cannot run a unit test with CMake that is meant to show a failure. Wow, what a brilliant system !

Right - I presume that there's a way around it if one has nothing else to do. I didn't find a simple one. This fact, combined with the fact that the system is well maintained, continually enhanced, and responsive to it's users for over 10 years says a lot the demands and state of library development. It seems almost no one ever asked kitware for this.

...

...

...

Does CMake have an equivalent to the Boost Build alias rule ? or the Boost Build project rule ?

LOL - I honestly don't remember what these rules do in boost build. But then I've only been using boost build for 15 years.

The project rule for unit tests is normally used to set usage requirements for all tests being run. The alias rule for unit tests is normally used as a means of grouping some unit tests together so that the end-user of the tests can test individual groups rather than all the tests.

My real point is that even one with a lot of experience with boost build (I would guess that the serialization library carpet bombing test suite is the most complex one in boost) still can't sit down and use it without spending huge amounts of time re-figuring it out. This is a serious indictment of boost build which has now brought us to this unhappy juncture.

Do you know if CMake has a way of setting usage requirements for all builds in a directory ? That is the effect of the project rule in Boost Build when running tests. It is very useful and means that you do not have to repeat a bunch of usage requirements for each and every test.

...

...

...

The great majority of the Boost libraries are header-only libraries where Boost Build is used to run the unit tests and,

true and this case is pretty simple in both cmake and boost build.

Once again saying it is pretty simple in cmake isn't showing me anything.

I've provide two working examples for two very different libraries and a narrative with the information I gleaned from making them. I would hope they would be helpful. It's the best anyone can/has done.

I will look at your serialization CMake code and I assume the other working example is for your Safe Numerics library, so I will look at that also. Thanks !

...

...

...

optionally, Boost Build is used to build the documentation using a quickbook, boostbook, doxygen toolset to generate html and pdf files.

Personally, I could never make this tool chain work despite a large investment of effort. I gave up on doxygen and quickbook after concluding them to be essentially non-functional. I did manage to use the xml tools loaded with boost tools along with some downloaded FOP, a version of XMLMind to create toolchain which is about 100 times better and 1/100 as complex as the boost one for creating documents. So you're life will be better when you're forced to leave these behind.

I heartily disagree. Using quickbook and doxygen is a no-brainer for me, but there is no point of debating this. Are both perfect ? No. But using them makes creating documentation supremely easy.

Right - we just have to disagree.

...

I did say that there is no Boost requirement for creating documentation using Boost Build. Some documentation is straight html. But for those libraries that do go the quickbook and doxygen way using Boost Build we need a CMake way of doing this if that is what the Boost Steering Committee is mandating.

Actually we don't. We can just use a small shell script or batch file. It's only about 5 lines long and it's about 20 times simpler than going into boost build trying to train an ant to train a flee. This is another think we'll just have to disagree on.

...

...

snip

...

...

But you might want to experiment a little with the following useful orthogonal ideas.

a) investigate how CMake Find<your library> should work and consider adding the files necessary to support it to your library. This wouldn't mess up any of your other work.

b) Consider making a simple CMake project to build library and run tests such as I did for the serialization library. CMake gurus it seems dismiss this effort. But it has worked for me. The main benefit for me is that permits to to build a working IDE which I can used to build and debug. For me, this has turned out to be a great time saver.

I do not know how you learned enough of CMake to do what you want but, apropos my RANT, I am finding my ability to do this severely hampered by the piecemeal documentation which CMake deems its own docs.

Ahh docs ... This is not a CMake issue. It's a software development issue. Documentation for software libraries and tools is generally atrocious. The thing that bugs me the most though is that developers don't agree with this statement. they think their stuff is pretty good an that the users just need to be smarter. They're dead wrong. They design crappy stuff then package then (under duress) write some docs which only serve to illustrate how lacking in coherence their original designs are in the first place. But then it's too late and everyone else is left with the job of shoveling a path through it.

I've got a lot to say about this and I will be giving a presentation on the subject at CPPCon 2017 for anyone who might be interested. That is the both of you.

I did not realize I was a split personality <g>.

Robert Ramey

On 7/27/2017 1:12 AM, Robert Ramey via Boost wrote:

On 7/26/17 7:28 PM, Edward Diener via Boost wrote:

...

But the case I brought up is that there is no "build' of the library, since it is header only, but the library needs to execute its tests using CMake. That appears to me to be different from your case.

Well, there is the safe numerics library which has CMake tests and is header only. It's much simpler than the serialization library.

https://github.com/robertramey/safe_numerics

You'll find a few CMakeLists.txt files in different places like include/example/test etc. Not that much unlike bjam where one has jam files in test, example, etc.

...

So you are saying that even when their is no actual library to be built, as is the case with a header only library, you still specify a CMakeLists.txt file as if you were building a library ?

yep - just like for bjam you specify a jam file in the test directory, for CMake you specifiy a CMakeLists.txt file in the test directory.

Great, thanks !

...

...

This is a serious short coming of CMake. Given this it's a mystery that CMake has been successful. But then then look at all the competition it's ever worse!!! To addressed this, I wrote a walk through in the boost library inclubator to help library developers get started with CMake. It uses on the safe numerics library CMake as an example. This example/narrative addresses only header-only files.

Would you please give a URL ?

https://github.com/robertramey/safe_numerics

I'm really disappointed you couldn't find this. It recently was reviewed for inclusion in boost and accepted!

You said you wrote a walk through in the boost library incubator and I asked for the URL of the walk through. You gave me a link to your safe numerics library on github. I have found http://blincubator.com/tools_cmak/. I will read this.

...

Do you know if CMake has a way of setting usage requirements for all builds in a directory ? That is the effect of the project rule in Boost Build when running tests. It is very useful and means that you do not have to repeat a bunch of usage requirements for each and every test.

Right. Every build system has to address this problem.

In bjam I can make a "rule" which passes "usage requirements".

You do not have to make a rule since the builtin "project" rule already does this for you.

In CMake I make a "function" which passes arguments.

So CMake has no way built-in command to say "here are the usage requirements for every target in this directory" other than to have to write a "function" each time ? That seems a bit poor.

Similar but different. For some reason every build system designer seems to fancy himself a language designer and what we get is some oddball/irregular "language" for describing the "build". When it barely works the release and then start patching it up to so it can be made to work. The result is what we have to today. An assortment of build languages none of which are very good. This is addressed by inventing a NEW build language which isn't really any better than the one it's meant to replace. This happens again and again - this is how we came to be where we are today.

...

I will look at your serialization CMake code and I assume the other working example is for your Safe Numerics library, so I will look at that also. Thanks !

There are other examples already in boost - made by people who know more about CMake than I do. (I hate wasting time on this kind of stuff). Unfortunately, many of these examples seem to be more elaborate than necessary and end up totally obscuring that which starts out simple. Its a chronic programmer disease to start with something understandable which works and then try to make it perfect there by mucking it up. Hollywood plastic surgeries have the exact same problem.

...

...

I've got a lot to say about this and I will be giving a presentation on the subject at CPPCon 2017 for anyone who might be interested. That is the both of you.

I did not realize I was a split personality <g>.

Hmmmm - It didn't occur to me that you might be going to CPPCon. So I guess I'll plan for three then.

No I don't go to CPPCon but if you can get programmers to create better documentation by your talk there I am all for it.

Robert

On 7/27/2017 8:12 AM, Florent Castelli via Boost wrote:

On 27/07/2017 10:24, Edward Diener via Boost wrote:

...

On 7/27/2017 1:12 AM, Robert Ramey via Boost wrote:

...

On 7/26/17 7:28 PM, Edward Diener via Boost wrote:

...

But the case I brought up is that there is no "build' of the library, since it is header only, but the library needs to execute its tests using CMake. That appears to me to be different from your case.

Well, there is the safe numerics library which has CMake tests and is header only. It's much simpler than the serialization library.

https://github.com/robertramey/safe_numerics

You'll find a few CMakeLists.txt files in different places like include/example/test etc. Not that much unlike bjam where one has jam files in test, example, etc.

...

So you are saying that even when their is no actual library to be built, as is the case with a header only library, you still specify a CMakeLists.txt file as if you were building a library ?

yep - just like for bjam you specify a jam file in the test directory, for CMake you specifiy a CMakeLists.txt file in the test directory.

Great, thanks !

...

...

...

This is a serious short coming of CMake. Given this it's a mystery that CMake has been successful. But then then look at all the competition it's ever worse!!! To addressed this, I wrote a walk through in the boost library inclubator to help library developers get started with CMake. It uses on the safe numerics library CMake as an example. This example/narrative addresses only header-only files.

Would you please give a URL ?

https://github.com/robertramey/safe_numerics

I'm really disappointed you couldn't find this. It recently was reviewed for inclusion in boost and accepted!

You said you wrote a walk through in the boost library incubator and I asked for the URL of the walk through. You gave me a link to your safe numerics library on github. I have found http://blincubator.com/tools_cmak/. I will read this.

...

...

Do you know if CMake has a way of setting usage requirements for all builds in a directory ? That is the effect of the project rule in Boost Build when running tests. It is very useful and means that you do not have to repeat a bunch of usage requirements for each and every test.

Right. Every build system has to address this problem.

In bjam I can make a "rule" which passes "usage requirements".

You do not have to make a rule since the builtin "project" rule already does this for you.

...

In CMake I make a "function" which passes arguments.

So CMake has no way built-in command to say "here are the usage requirements for every target in this directory" other than to have to write a "function" each time ? That seems a bit poor.

It does has some support for things like that, but it scales poorly. What you can do instead is have a target with all the requirements you want and link against it and they will bubble up. Or you just use a function, which is not a bad thing, they can remove a lot of boilerplate and make the intent clearer. Would you write ASM all the time? No, you write high level C++. A build system is no different.

What would be the CMake syntax for such a target ? In Boost Build the 'project' rule lets you set requirements. How would I do that with your target. Some CMake code please ?

...

...

Similar but different. For some reason every build system designer seems to fancy himself a language designer and what we get is some oddball/irregular "language" for describing the "build". When it barely works the release and then start patching it up to so it can be made to work. The result is what we have to today. An assortment of build languages none of which are very good. This is addressed by inventing a NEW build language which isn't really any better than the one it's meant to replace. This happens again and again - this is how we came to be where we are today.

...

I will look at your serialization CMake code and I assume the other working example is for your Safe Numerics library, so I will look at that also. Thanks !

There are other examples already in boost - made by people who know more about CMake than I do. (I hate wasting time on this kind of stuff). Unfortunately, many of these examples seem to be more elaborate than necessary and end up totally obscuring that which starts out simple. Its a chronic programmer disease to start with something understandable which works and then try to make it perfect there by mucking it up. Hollywood plastic surgeries have the exact same problem.

...

...

I've got a lot to say about this and I will be giving a presentation on the subject at CPPCon 2017 for anyone who might be interested. That is the both of you.

I did not realize I was a split personality <g>.

Hmmmm - It didn't occur to me that you might be going to CPPCon. So I guess I'll plan for three then.

No I don't go to CPPCon but if you can get programmers to create better documentation by your talk there I am all for it.

...

Robert

On 2017-07-27 17:08, Robert Ramey via Boost wrote:

...

No I don't go to CPPCon but if you can get programmers to create better documentation by your talk there I am all for it.

I'm sort of pessimistic:

a) I don't think that most developers think there is a problem. That is that the documentation they create is just fine. b) If there is a problem, it's that the users aren't smart enough c) Hence, I think that most developers will read the abstract and conclude that the subject is trivial and doesn't hold any value for them.

For what it's worth: - I know that my documentation is incomplete - I am amazed by what users figure out - I just read your abstract. I won't promise to be in the room (there is just too much going on at CppCon), but it sounds valuable to me. I sincerely hope that you touch on expression templates, too. Seems to me I need concepts and metaclasses to describe them. Cheers, Roland

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of John Maddock via Boost Sent: 27 July 2017 10:10 To: boost@lists.boost.org Cc: John Maddock Subject: Re: [boost] CMake and Boost Build tests

<snip>

...

...

optionally, Boost Build is used to build the documentation using a quickbook, boostbook, doxygen toolset to generate html and pdf files.

Personally, I could never make this tool chain work despite a large investment of effort. I gave up on doxygen and quickbook after concluding them to be essentially non-functional. I did manage to use the xml tools loaded with boost tools along with some downloaded FOP, a version of XMLMind to create toolchain which is about 100 times better and 1/100 as complex as the boost one for creating documents. So you're life will be better when you're forced to leave these behind.

Works for me.

I'm sorry you found them that way, Doxygen I hate

I also don't like the *look* of its readable output. But *Doxygen comment syntax for annotating classes, functions, macros etc works OK, and is the nearest thing to a Standard for annotating code, and not just C++.* These comments are readable in the code without cluttering and obscuring too much. Some examples of the *syntax*: File information: \brief This file is a partial reference implementation for the proposed "C++ binary fixed-point arithmetic" as specified in N3352.\n \details There is optional support for certain variations of fixed_point using preprocessor definitions. Not all of these are supported at the moment. The potential options include: Is supported now : \#define BOOST_FIXED_POINT_DISABLE_IOSTREAM\n Is supported now : \#define BOOST_FIXED_POINT_DISABLE_MULTIPRECISION\n Information above a class, here the constructor /*! Constructor for a data series to plot \tparam T an STL container: for example: @c multimap. \param begin Starting iterator into container of data series begin() to start at the beginning. \param end Ending iterator into container of data series, end() to finish with the last item. \param title Title for the plot. */ function svg_2d_plot_series& svg_2d_plot_series::line_on(bool on_) { //! Set @c true to draw line linking data points. line_style_.line_on_ = on_; return *this; //! \return Reference to svg_2d_plot_series to make chainable. Doxygen can now use the Clang compiler to extract all this into a form that other programs can then use to lay out in a human-readable (or other machine readable form). (Doxygen makes a good stab at template-heavy C++, but isn't a compiler). I believe that this is the way forward to producing documentation in the style we prefer (or in the style that others that others prefer).

(to be fair it's mostly the way folks use it rather than the tool itself),

+1 but Quickbook works extremely well for me + more than 1

As for the toolchain.... I find it amusing that this is the one thing folks most often complain about... even citing it as an example of why we shouldn't invent stuff ourselves... when it's actually the one thing that was *not* invented here!

It works for me - most of the trouble is with bjam syntax and dealing with MS file naming. Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 (0) 1539 561830

On Wed, 2017-07-26 at 22:48 -0400, Edward Diener via Boost wrote:

On 7/26/2017 3:42 PM, paul via Boost wrote:

...

On Wed, 2017-07-26 at 14:49 -0400, Edward Diener via Boost wrote:

...

Following John Maddock's appeal for practical solutions related to the move to CMake, I would like to know what the CMake equivalent is to the Boost Build unit test functionality.

In other words what do I write for CMake in order to do a Boost Build compile, compile-fail, link, link-fail, run, and run-fail unit tests ?

For a run test, you would do, something like:

add_executable(footest footest.cpp) add_test(NAME footest COMMAND $)

OK, that makes sense.

...

For a compile/link test:

add_library(footest STATIC EXCLUDE_FROM_ALL footest.cpp) add_test(NAME footest COMMAND ${CMAKE_COMMAND} --build . --target footest -- config $<CONFIGURATION> WORKING_DIRECTORY ${CMAKE_BINARY_DIR})

Why am I adding a library when all I am trying to do is see whether  source file(s) can be compiled to their object files successfully ( the  compile test ) or whether source file(s) can be compiled to their object  files successfully then linked as an executable without linking errors (  the link test ) ?

You have to add a target to be built, and then the test will invoke this target. Tests in cmake are just commands that can do anything, and in this case we instruct cmake to build a target. The reason its a library and not an executable is so that a `main` function is not needed.  Of course, this will test both compiling and linking together, as there is no separation for this in cmake.

...

To make the tests with expected failures, just set a property:

set_tests_properties(footest PROPERTIES WILL_FAIL TRUE)

OK, thanks !

...

You also can check for certain output, which nice for checking that the compile fail tests has triggered the static assert.

Ideally, the BCMTest module can make this simpler:

http://bcm.readthedocs.io/en/latest/src/BCMTest.html

Which you can write the equivalent tests like this:

Run test: bcm_test(NAME footest SOURCES footest.cpp)

Run fail test: bcm_test(NAME footest SOURCES footest.cpp WILL_FAIL)

Compile test: bcm_test(NAME footest SOURCES footest.cpp COMPILE_ONLY)

Compile fail test: bcm_test(NAME footest SOURCES footest.cpp COMPILE_ONLY WILL_FAIL)

Thanks !

...

...

Does CMake have an equivalent to the Boost Build alias rule ?

You can create interface target that includes the set of targets.

add_library(core INTERFACE) target_link_libraries(core INTERFACE im reader writer)

Thanks I will look at the use of INTERFACE.

...

...

or the Boost Build project rule ?

Hmm, I am not sure how this is relevant to cmake.

For the purposes of unit tests the project rule lets the developer set  usage requirements for all tests in the jamfile, rather than having to  repeat the usage requirements for each test. Since usage requirements in  Boost Build can be pretty specific depending on the compiler being used  the entire usage requirements for all tests can be pretty extensive.  Having to repeat that extensive list of usage requirements for every  test would be a PITA.

Well you can use the global function such as `add_compile_options` or `link_libraries` but these are discouraged as this will apply to every target created in the directory(and the usage requirements are not propagated). So in general using per-target is preferred.  However, in BCM, I have `bcm_link_libraries` that will link in those libraries when calling `bcm_test`, and can be excluded with the `NO_TEST_LIBS` option. So this provides the same convenience as the project rule. Although, a lot of cmake projects just write their own custom function to add the tests.

On Thu, 2017-07-27 at 11:19 -0400, Edward Diener via Boost wrote:

On 7/27/2017 10:45 AM, paul via Boost wrote:

...

On Wed, 2017-07-26 at 22:48 -0400, Edward Diener via Boost wrote:

...

On 7/26/2017 3:42 PM, paul via Boost wrote:

...

On Wed, 2017-07-26 at 14:49 -0400, Edward Diener via Boost wrote:

...

Following John Maddock's appeal for practical solutions related to the move to CMake, I would like to know what the CMake equivalent is to the Boost Build unit test functionality.

In other words what do I write for CMake in order to do a Boost Build compile, compile-fail, link, link-fail, run, and run-fail unit tests ?

For a run test, you would do, something like:

add_executable(footest footest.cpp) add_test(NAME footest COMMAND $)

OK, that makes sense.

...

For a compile/link test:

add_library(footest STATIC EXCLUDE_FROM_ALL footest.cpp) add_test(NAME footest COMMAND ${CMAKE_COMMAND} --build . --target footest -- config $<CONFIGURATION> WORKING_DIRECTORY ${CMAKE_BINARY_DIR})

Why am I adding a library when all I am trying to do is see whether source file(s) can be compiled to their object files successfully ( the compile test ) or whether source file(s) can be compiled to their object files successfully then linked as an executable without linking errors ( the link test ) ?

You have to add a target to be built, and then the test will invoke this target. Tests in cmake are just commands that can do anything, and in this case we instruct cmake to build a target. The reason its a library and not an executable is so that a `main` function is not needed.

But a 'compile' or 'link' rule always attempts at compiling or linking  an executable anyway, which has a 'main' function.

Yes, and it will still work with a main function, but I dont see the need for the main function. The compile-fail tests in the Fit library don't use a main function either.

...

Of course, this will test both compiling and linking together, as there is no separation for this in cmake.

...

...

To make the tests with expected failures, just set a property:

set_tests_properties(footest PROPERTIES WILL_FAIL TRUE)

OK, thanks !

...

You also can check for certain output, which nice for checking that the compile fail tests has triggered the static assert.

Ideally, the BCMTest module can make this simpler:

http://bcm.readthedocs.io/en/latest/src/BCMTest.html

Which you can write the equivalent tests like this:

Run test: bcm_test(NAME footest SOURCES footest.cpp)

Run fail test: bcm_test(NAME footest SOURCES footest.cpp WILL_FAIL)

Compile test: bcm_test(NAME footest SOURCES footest.cpp COMPILE_ONLY)

Compile fail test: bcm_test(NAME footest SOURCES footest.cpp COMPILE_ONLY WILL_FAIL)

Thanks !

...

...

Does CMake have an equivalent to the Boost Build alias rule ?

You can create interface target that includes the set of targets.

add_library(core INTERFACE) target_link_libraries(core INTERFACE im reader writer)

Thanks I will look at the use of INTERFACE.

...

...

or the Boost Build project rule ?

Hmm, I am not sure how this is relevant to cmake.

For the purposes of unit tests the project rule lets the developer set usage requirements for all tests in the jamfile, rather than having to repeat the usage requirements for each test. Since usage requirements in Boost Build can be pretty specific depending on the compiler being used the entire usage requirements for all tests can be pretty extensive. Having to repeat that extensive list of usage requirements for every test would be a PITA.

Well you can use the global function such as `add_compile_options` or `link_libraries` but these are discouraged as this will apply to every target created in the directory(and the usage requirements are not propagated).

That is actually what I want and is the equivalent of the Boost Build  'project' rule in a test jamfile with the 'requirements' options. Why  this is discouraged I would not know. Surely if you run a large number  of tests in a Boost Build test jamfile, it is much easier to specify  requirements for all test in one place rather than repeat them for each  and every test. This is especially true when requirements are based on  particular compilers possibly with particular versions of those compilers.

Yes, but a custom function is usually what is done to avoid repeating the requirements everywhere.

...

So in general using per-target is preferred.

However, in BCM, I have `bcm_link_libraries` that will link in those libraries when calling `bcm_test`, and can be excluded with the `NO_TEST_LIBS` option. So this provides the same convenience as the project rule. Although, a lot of cmake projects just write their own custom function to add the tests.

Where is your BCM ?

https://github.com/boost-cmake/bcm http://bcm.readthedocs.io

Is this going to be used by the CMake implementation  in Boost as mandated by the steering committee ?

I don't know. I think having a set cmake modules with common functionality will be necessary, and I am working on setting boost building and testing with these modules.

I am all in favor of  CMake code which makes the transition between Boost Build and CMake as  transparent as possible. If I, as a library developer, am expected to  provide CMake equivalents to my Boost Build unit tests and CMake  equivalents to my Boost Build doc generation I want as high a  boilerplate level as possible.

Yea, I think copying and pasting these solutions for every library is not maintainable, but we also need a balance. Originally, I had the BCM modules use a high-level `bcm_boost_package` to setup a library like this: bcm_boost_package(core     VERSION 1.61.0     DEPENDS         assert         config ) Which it would take care of a lot of cmake setup under the hood. The problem with this is it is unfamiliar to cmake developers and to non-cmake developers as well. So its kind of the worst of both worlds. So instead I am focusing on building functions that can reduce the boilerplate for boost-like tasks while still making the cmake as transparent as possible. So instead the above is written as: bcm_setup_version(VERSION 1.61.0) find_package(boost_assert) find_package(boost_config) add_library(boost_core INTERFACE) set_property(TARGET boost_core PROPERTY EXPORT_NAME core) target_link_libraries(boost_core INTERFACE boost::assert boost::config) bcm_deploy(TARGETS boost_core INCLUDE include NAMESPACE boost::) There could be more work to improve this, but I think this is a good balance.

On 7/27/2017 11:41 AM, Deniz Bahadir via Boost wrote:

Am 27.07.2017 um 16:45 schrieb paul via Boost:

...

On Wed, 2017-07-26 at 22:48 -0400, Edward Diener via Boost wrote:

...

On 7/26/2017 3:42 PM, paul via Boost wrote:

...

[...]

...

For a compile/link test:

add_library(footest STATIC EXCLUDE_FROM_ALL footest.cpp) add_test(NAME footest COMMAND ${CMAKE_COMMAND} --build . --target footest -- config $<CONFIGURATION> WORKING_DIRECTORY ${CMAKE_BINARY_DIR})

Why am I adding a library when all I am trying to do is see whether source file(s) can be compiled to their object files successfully ( the compile test ) or whether source file(s) can be compiled to their object files successfully then linked as an executable without linking errors ( the link test ) ?

You have to add a target to be built, and then the test will invoke this target. Tests in cmake are just commands that can do anything, and in this case we instruct cmake to build a target. The reason its a library and not an executable is so that a `main` function is not needed.

Of course, this will test both compiling and linking together, as there is no separation for this in cmake.

CMake has support for "object-libraries", which are targets that just compile source-files into object-files I have not tried it but I would think the following would be possible to only test compilation without linking:

```cmake

add_library(OBJECT footest EXCLUDE_FROM_ALL footest.cpp # ... further source-files can go here ... ) add_test(NAME footest COMMAND ${CMAKE_COMMAND} --build . --target footest --config $<CONFIGURATION> WORKING_DIRECTORY ${CMAKE_BINARY_DIR} )

Thanks, that looks reasonable for a compile test.

```

HTH, Deniz

...

For a run test, you would do, something like:

add_executable(footest footest.cpp) add_test(NAME footest COMMAND $)

"If <command> specifies an executable target (created by add_executable()) it will automatically be replaced by the location of the executable created at build time." So, add_test(NAME footest COMMAND footest)

On Jul 27, 2017 16:41, "Edward Diener via Boost" wrote: On 7/27/2017 8:43 AM, Florent Castelli via Boost wrote:

On 26/07/2017 20:49, Edward Diener via Boost wrote:

...

Following John Maddock's appeal for practical solutions related to the move to CMake, I would like to know what the CMake equivalent is to the Boost Build unit test functionality.

In other words what do I write for CMake in order to do a Boost Build compile, compile-fail, link, link-fail, run, and run-fail unit tests ?

In my own Boost-CMake project, I have implemented regular "RUN" in a way that looks similar to the original Boost Build using functions: https://github.com/Orphis/boost-cmake/blob/master/libs/system.cmake

What is RUN supposed to be in your link above ? An equivalent of run from Boost Build. See https://github.com/boostorg/system/blob/develop/test/Jamfile.v2 Other commands should be easy enough to implement as well, but I haven't

had the interest in doing that in my project yet (which predates the SC decision).

It is possible to do "compile" by creating a regular static library target with that file and then invoking "cmake --build . --target <compile test name>". Compile failures can be done similarly and check they fail. Link and Link-fail can be done the same way using actual binaries that are built, but not run.

This is a poor solution. Could you please explain why? Using Cmake directly to build the sources with the same flags as set in the toolchain is not controversial I'd say. It's actually working better than other solutions I've seen trying to reproduce the features.

The difference is making sure you have a clear separation of building Boost itself and running the tests. I found out that failures are pretty common with those tests and many of them will fail to run without errors on all supported compiler and platform, so isolation is quite important.

Again I need to mention that the tests are run for header only libraries which are not "built" at all in most cases. That's irrelevant for CMake. You end up building the tests still, they will depend on the header only library and its dependencies.

The problem is that some build tools will not support parallel invocations (Ninja doesn't for example, see https://groups.google.com/foru m/#!topic/ninja-build/4VP7whvWSH8 ) and thus running tests would be need to be linearized or moved to make, which should support that scenario better.

That is the end-users problem. I am just interested in Boost solving its own problem of moving from the Boost Build testing rules to something with CMake which does the same thing. Not really, because if a solution that works correctly and doing perfect forwarding of build flags can't be parallelized in some case, it would need fixing. If a solution doesn't have perfect forwarding of compiler settings and requirements, it's not really acceptable either.

...

Does CMake have an equivalent to the Boost Build alias rule ? or the Boost Build project rule ?

The great majority of the Boost libraries are header-only libraries where Boost Build is used to run the unit tests and, optionally, Boost Build is used to build the documentation using a quickbook, boostbook, doxygen toolset to generate html and pdf files. While I realize that building a Boost library, and making that built library available to end-users via CMake, has been the focus of most of the discussion so far regarding CMake and Boost, I am assuming the move to CMake also means that running unit tests and building docs where applicable must also be done through CMake. If I am wrong about this then I personally will be only too happy to continue using Boost Build to do these things.

CMake support header-only libraries using the INTERFACE keyword of "add_library()". Those will propagate usage requirements like any other library to their users. You can have you tests depend on those easily.

You can always have custom targets in each library to generate their documentation in any way you like. Building the documentation for the whole Boost project can be done by building all of those targets.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman /listinfo.cgi/boost

On 27 Jul 2017, at 22:29, Edward Diener via Boost wrote:

On 7/27/2017 12:58 PM, Florent Castelli via Boost wrote:

...

On Jul 27, 2017 16:41, "Edward Diener via Boost" wrote: On 7/27/2017 8:43 AM, Florent Castelli via Boost wrote:

...

On 26/07/2017 20:49, Edward Diener via Boost wrote:

...

Following John Maddock's appeal for practical solutions related to the move to CMake, I would like to know what the CMake equivalent is to the Boost Build unit test functionality.

In other words what do I write for CMake in order to do a Boost Build compile, compile-fail, link, link-fail, run, and run-fail unit tests ?

In my own Boost-CMake project, I have implemented regular "RUN" in a way that looks similar to the original Boost Build using functions: https://github.com/Orphis/boost-cmake/blob/master/libs/system.cmake

What is RUN supposed to be in your link above ? An equivalent of run from Boost Build. See https://github.com/boostorg/system/blob/develop/test/Jamfile.v2

Please show the code for RUN in CMake. To ease the transition from Boost Build to CMake we really need CMake equivalents to the unit testing rules of Boost Build at the very least.

Let’s start over. Basically, you end up with those equivalents. Run is compiling, linking an executable and running it, checking the return code is success. It could translate to: add_executable(test_name test_file.cpp) target_link_libraries(test_name PRIVATE boost_library) add_test(NAME test_name COMMAND test_name) But, if you do that, you need 2 steps to run the test. First you compile the program (using regular "make test_name" or even “cmake —build . —target test_name”) and then you run your test through ctest “ctest . -R test_name” (or you run all the tests without the filter). The problem there is you have 2 commands, and ctest won’t capture the compilation or linking errors. To work around that, you can use this instead: add_test(NAME test_name COMMAND cmake —build . —target test_name && $) This will for each test build it with the proper build tool (make, ninja, msbuild, xcodebuild…) and then run the program. The whole output will be captured by ctest and can be transfered to any test dashboard, you won’t have to look for errors in 2 places. The big issue I talk about later if that if you asked ctest to run several tests in parallel, it will have several “cmake —build . —target …” in parallel for those tests, which is not supported by some tools (mainly Ninja).

...

Other commands should be easy enough to implement as well, but I haven't

...

had the interest in doing that in my project yet (which predates the SC decision).

It is possible to do "compile" by creating a regular static library target with that file and then invoking "cmake --build . --target <compile test name>". Compile failures can be done similarly and check they fail. Link and Link-fail can be done the same way using actual binaries that are built, but not run.

This is a poor solution. Could you please explain why? Using Cmake directly to build the sources with the same flags as set in the toolchain is not controversial I'd say. It's actually working better than other solutions I've seen trying to reproduce the features.

I said it was a poor solution because all the Boost Build 'compile' rule does is to try to compile one or more source files. Success is if there are no compilations errors, while failure is if there is at least one compilation error. Why should one have to create an actual library to do this, especially as the source(s) may contain a main() function, and in fact usually does ?

I think the right solution, as suggested to me by someone else, is to create a CMake OBJECT library, which as I understand it is not really a library but just OBJECT files, and then run a test based on that. But of course I do not know if CMake supports that.

The thing is, static libraries are guaranteed to work and fairly standard, object libraries are a bit finicky and may not work depending on the generator. Using one or the other is merely an optimisation though, static libraries are just an archive of object files after all. Similar to the code above, you could implement them with this code: add_library(test_name STATIC (or OBJECT) test_file.cpp) target_link_libraries(test_name PRIVATE boost_library) add_test(NAME test_name COMMAND cmake —build . —target test_name) In this code, ctest will just build it and capture any compilation error. Easy enough. It doesn’t matter if there are functions called main or not since it’s not linked, the compiler won’t care for the most part about the name of the functions (well, main has some special semantics, but I doubt any test would rely on that). For completeness, here’s a possible implementation of “LINK”. I understand this one links the source files into a binary, but doesn’t run it. This is simple CMake code as well: add_executable(test_name test_file.cpp) add_test(NAME test_name COMMAND cmake —build . —target test_name) In this case, CMake will build and link the files from the ctest invocation. To support the failing variants for all the above, you need this set_test_properties(test_name PROPERTIES WILL_FAIL TRUE). Note that if you expect a test to fail running, but still succeed linking, that’s not directly supported by the code above and some changes need to be made. Overall, that’s a lot of boilerplate for each test, which is why you need some convenience functions to make the build files clean and simple, with a clear intent. In complex test cases, it will certainly be possible to have people using CMake commands directly, but that should be rare.

...

...

The difference is making sure you have a clear separation of building Boost itself and running the tests. I found out that failures are pretty common with those tests and many of them will fail to run without errors on all supported compiler and platform, so isolation is quite important.

Again I need to mention that the tests are run for header only libraries which are not "built" at all in most cases. That's irrelevant for CMake. You end up building the tests still, they will depend on the header only library and its dependencies.

...

The problem is that some build tools will not support parallel invocations (Ninja doesn't for example, see https://groups.google.com/foru m/#!topic/ninja-build/4VP7whvWSH8 ) and thus running tests would be need to be linearized or moved to make, which should support that scenario better.

That is the end-users problem. I am just interested in Boost solving its own problem of moving from the Boost Build testing rules to something with CMake which does the same thing. Not really, because if a solution that works correctly and doing perfect forwarding of build flags can't be parallelized in some case, it would need fixing. If a solution doesn't have perfect forwarding of compiler settings and requirements, it's not really acceptable either.

Why would not each test have the correct flags whether the tests run in parallel or not ? Why would you want tests to run in parallel and not be linearized ?

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost http://lists.boost.org/mailman/listinfo.cgi/boost

On Thu, 2017-07-27 at 17:53 -0400, Edward Diener via Boost wrote:

On 7/27/2017 5:15 PM, Florent Castelli via Boost wrote:

...

...

On 27 Jul 2017, at 22:29, Edward Diener via Boost

...

wrote:

On 7/27/2017 12:58 PM, Florent Castelli via Boost wrote:

...

On Jul 27, 2017 16:41, "Edward Diener via Boost" wrote: On 7/27/2017 8:43 AM, Florent Castelli via Boost wrote:

...

On 26/07/2017 20:49, Edward Diener via Boost wrote:

...

Following John Maddock's appeal for practical solutions related to the move to CMake, I would like to know what the CMake equivalent is to the Boost Build unit test functionality.

In other words what do I write for CMake in order to do a Boost Build compile, compile-fail, link, link-fail, run, and run-fail unit tests ?

In my own Boost-CMake project, I have implemented regular "RUN" in a way that looks similar to the original Boost Build using functions: https://github.com/Orphis/boost-cmake/blob/master/libs/system.cmake

What is RUN supposed to be in your link above ? An equivalent of run from Boost Build. See https://github.com/boostorg/system/blob/develop/test/Jamfile.v2

Please show the code for RUN in CMake. To ease the transition from Boost Build to CMake we really need CMake equivalents to the unit testing rules of Boost Build at the very least.

Let’s start over. Basically, you end up with those equivalents. Run is compiling, linking an executable and running it, checking the return code is success. It could translate to:

add_executable(test_name test_file.cpp) target_link_libraries(test_name PRIVATE boost_library) add_test(NAME test_name COMMAND test_name)

But, if you do that, you need 2 steps to run the test. First you compile the program (using regular "make test_name" or even “cmake —build . —target test_name”) and then you run your test through ctest “ctest . -R test_name” (or you run all the tests without the filter). The problem there is you have 2 commands, and ctest won’t capture the compilation or linking errors. To work around that, you can use this instead:

add_test(NAME test_name COMMAND cmake —build . —target test_name && $)

This will for each test build it with the proper build tool (make, ninja, msbuild, xcodebuild…) and then run the program. The whole output will be captured by ctest and can be transfered to any test dashboard, you won’t have to look for errors in 2 places. The big issue I talk about later if that if you asked ctest to run several tests in parallel, it will have several “cmake —build . —target …” in parallel for those tests, which is not supported by some tools (mainly Ninja).

Understood.

...

...

...

Other commands should be easy enough to implement as well, but I haven't

...

had the interest in doing that in my project yet (which predates the SC decision).

It is possible to do "compile" by creating a regular static library target with that file and then invoking "cmake --build . --target <compile test name>". Compile failures can be done similarly and check they fail. Link and Link-fail can be done the same way using actual binaries that are built, but not run.

This is a poor solution. Could you please explain why? Using Cmake directly to build the sources with the same flags as set in the toolchain is not controversial I'd say. It's actually working better than other solutions I've seen trying to reproduce the features.

I said it was a poor solution because all the Boost Build 'compile' rule does is to try to compile one or more source files. Success is if there are no compilations errors, while failure is if there is at least one compilation error. Why should one have to create an actual library to do this, especially as the source(s) may contain a main() function, and in fact usually does ?

I think the right solution, as suggested to me by someone else, is to create a CMake OBJECT library, which as I understand it is not really a library but just OBJECT files, and then run a test based on that. But of course I do not know if CMake supports that.

The thing is, static libraries are guaranteed to work and fairly standard, object libraries are a bit finicky and may not work depending on the generator. Using one or the other is merely an optimisation though, static libraries are just an archive of object files after all. Similar to the code above, you could implement them with this code:

add_library(test_name STATIC (or OBJECT) test_file.cpp) target_link_libraries(test_name PRIVATE boost_library) add_test(NAME test_name COMMAND cmake —build . —target test_name)

In this code, ctest will just build it and capture any compilation error. Easy enough. It doesn’t matter if there are functions called main or not since it’s not linked, the compiler won’t care for the most part about the name of the functions (well, main has some special semantics, but I doubt any test would rely on that).

But now you are filling up an end-user's build output with a static  library for each 'compile' tested. Ughhh ! That seems so silly to me,  where just adding object files on success seems normal.

Well actually it should be added with `EXCLUDE_FROM_ALL`: add_library(test_name STATIC test_file.cpp EXCLUDE_FROM_ALL) This way it only builds when it is invoked directly(which is when the test is ran), and not in the general build.

On 27/07/2017 23:53, Edward Diener via Boost wrote:

On 7/27/2017 5:15 PM, Florent Castelli via Boost wrote:

...

...

On 27 Jul 2017, at 22:29, Edward Diener via Boost wrote:

On 7/27/2017 12:58 PM, Florent Castelli via Boost wrote:

...

On Jul 27, 2017 16:41, "Edward Diener via Boost" wrote: On 7/27/2017 8:43 AM, Florent Castelli via Boost wrote:

...

On 26/07/2017 20:49, Edward Diener via Boost wrote:

...

Following John Maddock's appeal for practical solutions related to the move to CMake, I would like to know what the CMake equivalent is to the Boost Build unit test functionality.

In other words what do I write for CMake in order to do a Boost Build compile, compile-fail, link, link-fail, run, and run-fail unit tests ?

In my own Boost-CMake project, I have implemented regular "RUN" in a way that looks similar to the original Boost Build using functions: https://github.com/Orphis/boost-cmake/blob/master/libs/system.cmake

What is RUN supposed to be in your link above ? An equivalent of run from Boost Build. See https://github.com/boostorg/system/blob/develop/test/Jamfile.v2

Please show the code for RUN in CMake. To ease the transition from Boost Build to CMake we really need CMake equivalents to the unit testing rules of Boost Build at the very least.

Let’s start over. Basically, you end up with those equivalents. Run is compiling, linking an executable and running it, checking the return code is success. It could translate to:

add_executable(test_name test_file.cpp) target_link_libraries(test_name PRIVATE boost_library) add_test(NAME test_name COMMAND test_name)

But, if you do that, you need 2 steps to run the test. First you compile the program (using regular "make test_name" or even “cmake —build . —target test_name”) and then you run your test through ctest “ctest . -R test_name” (or you run all the tests without the filter). The problem there is you have 2 commands, and ctest won’t capture the compilation or linking errors. To work around that, you can use this instead:

add_test(NAME test_name COMMAND cmake —build . —target test_name && $)

This will for each test build it with the proper build tool (make, ninja, msbuild, xcodebuild…) and then run the program. The whole output will be captured by ctest and can be transfered to any test dashboard, you won’t have to look for errors in 2 places. The big issue I talk about later if that if you asked ctest to run several tests in parallel, it will have several “cmake —build . —target …” in parallel for those tests, which is not supported by some tools (mainly Ninja).

Understood.

...

...

...

Other commands should be easy enough to implement as well, but I haven't

...

had the interest in doing that in my project yet (which predates the SC decision).

It is possible to do "compile" by creating a regular static library target with that file and then invoking "cmake --build . --target <compile test name>". Compile failures can be done similarly and check they fail. Link and Link-fail can be done the same way using actual binaries that are built, but not run.

This is a poor solution. Could you please explain why? Using Cmake directly to build the sources with the same flags as set in the toolchain is not controversial I'd say. It's actually working better than other solutions I've seen trying to reproduce the features.

I said it was a poor solution because all the Boost Build 'compile' rule does is to try to compile one or more source files. Success is if there are no compilations errors, while failure is if there is at least one compilation error. Why should one have to create an actual library to do this, especially as the source(s) may contain a main() function, and in fact usually does ?

I think the right solution, as suggested to me by someone else, is to create a CMake OBJECT library, which as I understand it is not really a library but just OBJECT files, and then run a test based on that. But of course I do not know if CMake supports that.

The thing is, static libraries are guaranteed to work and fairly standard, object libraries are a bit finicky and may not work depending on the generator. Using one or the other is merely an optimisation though, static libraries are just an archive of object files after all. Similar to the code above, you could implement them with this code:

add_library(test_name STATIC (or OBJECT) test_file.cpp) target_link_libraries(test_name PRIVATE boost_library) add_test(NAME test_name COMMAND cmake —build . —target test_name)

In this code, ctest will just build it and capture any compilation error. Easy enough. It doesn’t matter if there are functions called main or not since it’s not linked, the compiler won’t care for the most part about the name of the functions (well, main has some special semantics, but I doubt any test would rely on that).

But now you are filling up an end-user's build output with a static library for each 'compile' tested. Ughhh ! That seems so silly to me, where just adding object files on success seems normal.

I understand that, but some IDEs may have issues with those. I remember having issues with Xcode a while ago. Requires more testing I would say.

...

For completeness, here’s a possible implementation of “LINK”. I understand this one links the source files into a binary, but doesn’t run it. This is simple CMake code as well:

add_executable(test_name test_file.cpp) add_test(NAME test_name COMMAND cmake —build . —target test_name)

In this case, CMake will build and link the files from the ctest invocation.

To support the failing variants for all the above, you need this set_test_properties(test_name PROPERTIES WILL_FAIL TRUE). Note that if you expect a test to fail running, but still succeed linking, that’s not directly supported by the code above and some changes need to be made.

Overall, that’s a lot of boilerplate for each test, which is why you need some convenience functions to make the build files clean and simple, with a clear intent. In complex test cases, it will certainly be possible to have people using CMake commands directly, but that should be rare.

Understood. Thanks !

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost