Re: [boost] Boost CMake support - Request for Comment

On 10/9/2018 7:09 PM, mike via Boost wrote:

...

-----Original Message----- From: Boost On Behalf Of Edward Diener via Boost Sent: Monday, October 8, 2018 10:22 PM

On 10/8/2018 6:15 AM, Mike Dev via Boost wrote:

...

...

-----Original Message----- From: Boost On Behalf Of Edward Diener via Boost Sent: Monday, October 1, 2018 8:00 PM

On 10/1/2018 1:07 PM, Robert Ramey via Boost wrote:

...

Background ==========

My personal knowledge of build systems is somewhat limited.

a) I don't really understand CMake all that well. I have made CMakeLists.txt files for both the serialization library and the safe numerics libraries as an aid to build and maintain IDE and make files used in running tests and posting results to CDash.

b) I don't really understand our current system, Boost Build, very well either. I've made jam files for building and running tests for the serialization and safe numerics libraries.

c) I recently reviewed C++Now 2017: Daniel Pfeifer's excellent video from C++Now 2017 “Effective CMake" - https://www.youtube.com/watch?v=bsXLMQ6WgIk . I recommend it highly.

Despite my lack of knowledge in this area, or perhaps because of it, I've volunteered to direct the effort to evolve Boost so that users of Boost can avail themselves of the benefits of CMake. This effort will be organized as follows:

a) A discussion on the boost developers mailing list with the goal of reaching a consensus as to what benefits CMake can and should provide to users and developers of Boost. This effort will commence with the posting of this notice. I hope that this discussion can be more or less resolved within 30 days.

b) I will then synthesize from the above discussion a call for proposals which lists the requirements and requirements.

c) Those proposing CMake for boost will have approximately 90 days to prepare their proposals. Their proposal are expected to look similar to a boost library proposal. That is they are expected to have some (CMake) code, along with tests, documentation including tutorials and reference. This is only an expectation. It's conceivable that such a proposal might only contain conventions and and samples of what boost library should contain to fulfill the requirements. As with boost libraries, submissions are not required to totally complete. But they have to be sufficiently complete to convince reviewers the the submission fulfills the stated requirements and can be finished in good time.

d) After the 90 day period, submissions will be reviewed simultaneously. Note that this is at a variance from normal boost procedure of reviewing submissions one at a time. Since we expect to receive a number of submissions and only one can be accepted, the normal boost protocol can't really function. Target date for this review is 1 February 2019.

e) After the review period the review manager will prepare a report which includes the decision of which submission will be accepted into the official boost distribution. Currently, I, Robert Ramey, expect to be review manager. However, it's possible that this by the time the review is undertaken, this coveted position could be assigned to some more worthy candidate.

Scope, Requirements and Features ================================

We presume that submissions will fulfill the applicable requirements of any boost submission. In addition we would like to see the following addressed.

Library Build ------------- “Effective CMake" - https://www.youtube.com/watch?v=bsXLMQ6WgIk Recommends that all libraries - including header only libraries have a CMakeList.txt file. What should this include for Boost Libraries?

Library Test ------------ Should facilities for "testing" be only done by developers? or should users who acquire library packages also be able to test libraries. Should CMake testing results posting be used - CDASH? Should we just skip the issue of Library Testing and continue to depend on Boost Build.

Library Packaging ----------------- Is the library packaging facility provide by CMake - CPACK - useful to boost. Should boost libraries be updated to support it?

Dependencies ------------ If the above is implemented, can we depend upon CMake to handle library dependencies? Or does some special functionality Do circular dependencies constitute a problem?

Modularity ---------- Currently boost is organized as a tightly integrated group of libraries. This organization manifests itself in a number of ways. We have a Boost "super project" in github. Boost libraries are "sub projects" of this super project. This known structure is exploited by and depended upon by boost tools. Should CMake support continue this policy/design. We distribute boost as "super project". Should we continue to do this or distribute libraries on a library by library basis. Does CMake/CPACK etc. make this "simple"?

Documentation/testing --------------------- What support should be require for uses of Boost CMake implementation. Documentation, examples, templates for helping library developers. How should the CMake design/implementation be tested?

Other considerations -------------------- add your requirement/features here.

Useful Resources =============== “Effective CMake" - https://www.youtube.com/watch?v=bsXLMQ6WgIk https://gist.github.com/mbinna/c61dbb39bca0e4fb7d1f73b0d66a4fd1 https://github.com/purpleKarrot/Boost.CMake https://www.youtube.com/watch?v=eC9-iRN2b04

Please try to keep this discussion on this list.

I know that there has been a lot going on on slack. But having it in two places makes it harder for me to review and summarize. Also slack doesn't conveniently maintain comment hierarchy. So please try to keep this discussion on this list.

I strongly suggest that the goal of using CMake with Boost begin with making Boost libraries, whether header-only or built, available to CMake projects. This also means that we have a way of testing whether or not such a solution actually works for any given Boost library. My own preference for testing whether or not such a solution would work or not would be to use our current b2/Boost Build test jamfiles, but if this is too difficult to do we still need some way to test the solution for each library so each library maintainer can see whether or not the solution we choose works for his library.

This should be our first goal and all other goals should be tabled until and when we can have a solution where this works flawlessly and there is a general agreement that whatever proposal is chosen to implement this is best.

Only when the above is fully accomplished should Boost then look into the goal of using CMake internally, perhaps for test, docs, building libraries, or what-not. We need to work this way because every solution is bound to end with comments that it does X but does not do Y or it does X and Y but does not do Z, and we all know where these endless arguments end: that we end up doing nothing because we start by trying to do too much and we can never reach agreement.

I agree with you in so far, that the focus should be on the interface (how to consume boost libraries) and less on the implementation details, but I see two practical reasons, why it is beneficial to immediately use cmake (at least) for building too:

1) Creating a CMake target for a separately compiled binary that can be consumed by the user is about the same effort as using cmake to build the library (if necessary) in which case you get the interface information for free (switching to cmake for unit-tests and docs is a different topic).

2) "making Boost libraries [...] available to CMake projects" for me also means I have an easy way to compile those libraries with specific compiler flags that match my project settings - otherwise, what is the advantage compared to the current mechanism that is already build into cmake? Ensuring consistent build flags is relatively easy if all libraries use the same build system (in this case cmake) but can become quite complex if you want to implement an automatic translation from cmake to b2 or vice versa.

So yes, while a library (build-) interface and consummation should be the focus of the review, I'd expect that at least building the libraries with cmake will be the simplest way to implement that interface.

Just to be completely clear: Just because the "reference implementation" of a cmake interface is probably a cmake-based build, that doesn't mean everyone has to actually adopt that reference implementation as long as the interface is consistent.

Boost Build has a feature which allows the programmer to test at build time what C++ features are supported for a particular compiler implementation and compile parameters, and change the build internally on-the-fly accordingly. I do not know if any Boost non-header only library uses this feature but if one does it will be difficult translating this to CMake terms, if such a similar feature exists in CMake.

As Richard pointed out, such features exist in cmake too and have been in common use for years. E.g. It has a built-in mechanism that lets you directly query for most of the post c++03 compiler features: (https://cmake.org/cmake/help/v3.13/prop_gbl/CMAKE_CXX_KNOWN_FEATURES.html) and as a more general feature, you can always check if a particular source file using some feature or type compiles or not: (https://cmake.org/cmake/help/v3.13/module/CheckCXXSourceCompiles.html)

Could you maybe point me to some examples where this feature is used so one could estimate how difficult a conversion is?

The feature is essentially programmed by config and predef to provide build-time testing of compiler support and testing of such things like the OS versions and compiler versions etc. See https://www.boost.org/doc/libs/1_68_0/libs/config/doc/html/boost_config/buil... for the explanation in config and https://www.boost.org/doc/libs/1_68_0/doc/html/predef/check_utilities.html for checking predef values at build time. The basics of the feature are offered by Boost Build as explained at https://boostorg.github.io/build/manual/develop/index.html#bbv2.reference.ru... in reference to the check-target-builds rule. Plenty of libraries use the feature during testing and I imagine that some of them may use the feature during building. Asking individual libraries to convert their tests or build to use some CMake equivalent, for every possible setting in config and predef which they use, is not realistic. I understand that CMake may have an equivalent technique, but who is going to program that technique to offer all the equivalents which config and predef currently offer ?

...

It is for such a reason that as soon as you start talking about using CMake for anything else but identifying Boost libraries for the purposes of end-user use of Boost libraries in their own CMake scripts,

Just consuming precompiled/pre-installed boost libraries in a cmake script is already possible thanks to the work of cmake contributors and the people providing boost packages for vcpkg, conan and other package management systems (some of those contributors are probably also boost library maintainers?). Those solutions have to keep playing catch up with the latest boost release if something significant changes (like name-mangling or a new library), but by and large they work quite well. Imho an official boost solution should provide a bit more functionality to justify the effort.

...

that you may run into much more work than you can imagine. You just can't tell individual library developers to translate all their own use of Boost Build into CMake, as if this will be simple for them to do, and they will just do it with little time spent and as if it will be no work for them.

Let's get one thing straight: Switching to cmake will require work, no one is denying that and it will not happen during the course of a single release. And if library developers are not willing to invest some time to do this, we can forget about it all together. However, it is my impression that many developers are willing to make that effort. All that is lacking is a clear decision of what the (semi-) final solution should look like to make sure the effort isn't wasted.

Now as I said. I'm not convinced, that only providing interface information is actually less work than building boost with cmake and letting it generate the cmake information automatically. That probably depends to a large degree if those cmake files can be auto generated by b2. However, just as there seem to be more complex build scenarios, there are probably scenarios where the auto-generated cmake-config files are not enough.

...

Boost Build, for better or worse, has some extensive features which have no easy correspondence to CMake.

Such as? If plain cmake is lacking any important features that are needed by boost, then a replacement is exactly what a boost cmake library should provide.

...

Unless you or someone else is willing to develop a bridge between Boost Build and CMake for features of Boost Build which libraries rely on, then individual Boost libraries which use such Boost Build features are going to find it very difficult to transition to CMake, possibly even for something as seemingly innocuous as building the library itself using a CMake script.

Let's not forget, that there are already solutions out there that build (almost?) all of boost with cmake. Here is one: https://github.com/boost-cmake/boost

Now instead of adopting and tweaking one of those existing solutions during review, you are imho asking for the development of yet another solution. I'm not sure if that is actually an easier path forward. That doesn't mean that an individual library author can't decide he wants to stick to b2 internally and just provide a cmake interface file, but we for the time being and

...

This is why I have urged the move to CMake by Boost to proceed slowly by baby steps, rather than to try doing things that may take a huge initial effort to get done, and therefore may end up never accomplishing anything.

That is kind of what I'm trying to achieve with my individual mini PRs. They are in no way a replacement for the existing boost build system, as they only cover a very narrow use case, but even those also build the boost libraries. Again, I don't think building is the issue (as I said, it has been done already) and if someone really encounters a problem, I'm sure people on the ML and elsewhere are willing to help. IMHO it will be far more difficult to agree all the bike shedding topics related to how exactly boost libraries should be made available to the consumer.

Also, proceeding by Baby steps means that boost will stay in limbo between cmake and boost build for quite some time, where library maintainers will have to maintain two systems and more or less constantly upgrading their CMake files. It doesn't have to happen all in one big step (I agree that that is not realistic), but considering that consumers can already use boost libraries in their cmake projects (albeit with a few pain points here and there) and that there are already proof of concepts for more fully fledged solutions out there, I think the initial goal should be a tad more ambitious.

Mike