Glen Fernandes wrote:

On Sun, Jun 6, 2021 at 9:12 PM Peter Dimov via Boost wrote:

...

Some libraries (I encountered this when porting the tests of Signals2) use Boost.Test in header-only mode, by including without linking to anything. This worked in the "legacy" structure where all headers are always available, but doesn't work under CMake where you have to link to header-only libraries in order to get the proper include path.

Should we make an attempt to support this use by declaring a special header-only target in Boost.Test (e.g. Boost::included_unit_test), or is that a waste of time? One can as well link to the proper Boost::unit_test_framework (and then even use it header-only, although there's not much point in doing so.)

I'm inclined towards "waste of time".

Likewise.

What is the list of libraries that do this? I'm curious if most of them are unmaintained or would otherwise be better served migrating to LWT.

The libraries that use the "included/" versions of the Test libraries are actually the maintained ones, because the unmaintained ones are still on the deprecated . :-) The `included` ones are algorithm filesystem geometry heap lockfree math mpi numeric/conversion numeric/odeint numeric/ublas process ptr_container random range signals2 sort type_traits uuid yap so this is widely used, and in fairness, it's a legitimate and documented way to use Boost.Test. I don't think we need to change anything in the libraries, but I also don't think we need to duplicate the Boost.Test CMake targets to make them have header-only versions. Linking to the normal target will also enable the header-only use to compile. For completeness, the libraries using the deprecated boost/test/minimal.hpp are: foreach graph graph_parallel iterator logic numeric/conversion numeric/interval property_map tokenizer units yap These need to migrate to either LWT or the non-deprecated Boost.Test.

On 07/06/2021 17:26, Peter Dimov via Boost wrote:

Edward Diener wrote:

...

Since it is totally legitimate to use Boost.Test in header only mode, and many libraries currently do this, it is hardly a "waste of time" to support this under CMake. Under CMake you can't use header-only libraries without linking to their targets. So there's no real benefit from having a separate header-only target to link to; you can just link to the ordinary target Boost::unit_test_framework.

Doesn't that a) cause the unit test lib to be build when it doesn't need to be, and b) add the resulting binary to library link list when it doesn't need to be? But I do appreciate the simplicity of a single target ;) John. -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Andrey Semashev wrote:

On 6/7/21 7:48 PM, John Maddock via Boost wrote:

...

On 07/06/2021 17:26, Peter Dimov via Boost wrote:

...

Edward Diener wrote:

...

Since it is totally legitimate to use Boost.Test in header only mode, and many libraries currently do this, it is hardly a "waste of time" to support this under CMake.

...

...

Under CMake you can't use header-only libraries without linking to their targets. So there's no real benefit from having a separate header-only target to link to; you can just link to the ordinary target Boost::unit_test_framework.

Doesn't that a) cause the unit test lib to be build when it doesn't need to be, and b) add the resulting binary to library link list when it doesn't need to be?

But I do appreciate the simplicity of a single target ;)

Another question is whether unnecessarily linking would cause duplicate symbols or ODR issues. I believe, you're not supposed to use header-only Boost.Test and then link against it, too, as the same symbols would be defined both in the test and the library.

It works because the symbols in the executable override those in libraries, but it's indeed not quite right in principle.

Boost.Test could define two sets of targets - one for header-only and the other for separately built library. One benefit of that is that you could use different macros for these two targets, if needed (e.g. if the library sources need to discriminate between the two configs).

That's probably what we'll have to do, yes. Any input from the Boost.Test maintainer? Raffi?

Raffi Enficiaud wrote:

I have not looked at the recent developments for CMake in Boost. If the CMakeLists.txt I developed (under build/) is still there, it contained 2 targets: one for static and one for shared. Those were having different and explicit names.

I decided to not do this and instead use the "normal" CMake way of relying on BUILD_SHARED_LIBS. It causes problems in the more complicated cases where a compiled Boost::X links to header-only Boost::Y which links to a compiled Boost::Z. Client code can choose to link explicitly to Boost::static_X, and if X used Z directly, it could link to Boost::static_Z, but it doesn't. And requiring all header-only libraries to also provide triplicate targets so that static/shared-ness can be propagated correctly is (I think) infeasible. So X links to Boost::Y, and Y has to choose whether to link to the static or shared Z, but it doesn't know how. (b2 does this "right" by having static/shared as a build feature, and propagating it automatically. No need to split your targets by hand.) Swimming upstream is sometimes justified when it results in something objectively better than the accepted CMake way, but I don't believe that's the case here, so BUILD_SHARED_LIBS will have to do. (Similarly, I tried to do something b2-like with the tests, but at the end reverting to the way CMake wants to do things proved better.) Getting back to Boost.Test, I've duplicated the targets with an included_ prefix: https://github.com/boostorg/test/commit/bce2d24c8b32f47f0403766fe4fee3e2e93a...

On 8/06/2021 4:53 pm, Edward Diener wrote:

I do not know CMake, so maybe my comment is irrelevant, but it seems natural to me that a header-only library would always choose to use some other dependent library as header-only, even when that other library had static or shared variants. In other words I applaud the decision of Boost.Test to provide a header-only variant and can not even begin to process the fact that CMake can not deal with Boost.Test as a header-only library, if that is indeed the case.

A problem occurs when program P uses library A as static or shared and then wants to use header-only library B which also uses library A. For this to work, one of the following must be true: 1. library B can figure out how program P chose to link to it and links to it the exact same way. 2. library A is written such that in header-only mode it uses entirely separate namespaces from its compiled modes; then B can link to it in header-only mode without conflicting with other uses. #2 is rarely true, because it's more work for both users and maintainers (inline namespaces make it actually *feasible*, but still require people to actually have considered the problem in the first place). #1 is putting the cart before the horse, which can be problematic (but nevertheless, is the "solution" most often taken, by "bubbling down" settings from the parent projects) -- but it only works as long as everyone respects them and everything gets compiled with matching settings. (#1 also affect dependencies on the CRT itself, which is why most have big global hammers to specify whether linking to the CRT statically or dynamically, and why everything explodes when these settings don't match between compilation units.) When these conditions are not met, you end up with an ODR violation. Which often works anyway by coincidence, especially in libraries without singletons, but you're tap-dancing over a minefield. (And even doing #2 will land you with disjoint singleton "islands", but at least that way it's presumably deliberate.) (The safest way out of this minefield is to ensure that header-only libraries either have no separate dependencies, or at least only ever reference dependencies that do not have a compiled variant. This is of course a scaling problem.) TLDR: C++ is extremely ODR-prone and libraries are very fragile things.

On 8/06/2021 7:42 pm, I wrote:

Another way of putting all this is that libraries that can be configured between static/shared are mostly fine these days, because the tools have some blunt hammers that mostly do the right things with them.  (But still bite people on occasion.)

FWIW, the most common biting issue is when mixing static/shared. All-static and all-shared are usually pretty safe. But when libraries A and B both use dependency C, and either of A and B is shared, then C must be shared as well -- things will often break horribly if C is static. (Having said that, what actually happens is highly dependent on whether using Windows or something UNIXy, and for the latter what symbol visibility level is being used. And for some libraries and some platforms/settings then there will be no apparent problems.) But this sort of thing is usually why tools like CMake prefer to offer global settings to configure static/shared rather than letting it be configured on a target-by-target basis. You *can* override that, but it is at your own peril. (Or that of the downstream app.)

On 11/06/2021 6:55 pm, Alexander Grund wrote:

Of course this still allows to build your Boost::foo explicitely as static or shared in which case the problem is also avoided as then only one version of it exists anyway.

No, if you choose to build Boost::foo as static and it happens to be used by at least two different consumers (at least one of which is shared) that end up as part of a single final application, that's exactly the problem case that I was talking about. This is because that can end up with multiple copies of the library in the same application (but in separate binaries), subject to the whims of the platform and linker.

On 11/06/2021 7:52 pm, Alexander Grund wrote:

I meant the case where Boost::foo is ONLY ever build as static. I.e. `add_library(Boost::Foo STATIC source.cpp)` as opposed to `add_library(Boost::Foo source.cpp)` in CMake terms.

So there is only static Boost::Foo and no shared version EVER. This would work, wouldn't it?

Only if the library doesn't have any singletons, or otherwise can cope with multiple independent "islands".

On 6/9/21 2:05 PM, Rainer Deyke via Boost wrote:

On 08.06.21 09:33, Gavin Lambert via Boost wrote:

...

On 8/06/2021 4:53 pm, Edward Diener wrote:

...

I do not know CMake, so maybe my comment is irrelevant, but it seems natural to me that a header-only library would always choose to use some other dependent library as header-only, even when that other library had static or shared variants. In other words I applaud the decision of Boost.Test to provide a header-only variant and can not even begin to process the fact that CMake can not deal with Boost.Test as a header-only library, if that is indeed the case.

A problem occurs when program P uses library A as static or shared and then wants to use header-only library B which also uses library A.

For this to work, one of the following must be true:    1. library B can figure out how program P chose to link to it and links to it the exact same way.

A header-only library cannot *link* to anything, or it is not header-only.

CMake defines targets for header-only libraries, which can have dependencies on other targets, including on static or shared libraries. So the program that "links" to the header-only library picks up its dependencies recursively. As a result, the linker is invoked with all (binary) libraries in the dependency tree. Whether you can consider such library header-only in the first place is a philosophical question. I do, because otherwise any library that calls the standard library cannot be called header-only.

A header-only library B can depend on symbols being provided by the calling program P, which can delegate the task to library A, or to another library C which provides the same interface as A.  There should be no direct connection from B to A.  Depending on the design of B, this may involve #including a header from A before #including a header from B, or it can involve making the headers from A available to B so that B can #include them directly.  Where these symbols and/or headers come from is the business of P, not of B.

Reverse dependencies are rather uncommon, and it's not what is being discussed here. The question is how the header-only library selects the targets to depend on. I think, it should not select any specific target unless it absolutely must (i.e. it won't work otherwise). In terms of the earlier example, the library B should depend on the generic target A. Now, that target A can be defined as a header-only, shared or static variant, depending on the user's choice or some default.

Am 11.06.21 um 11:17 schrieb Rainer Deyke via Boost:

On 10.06.21 23:36, Andrey Semashev via Boost wrote:

...

On 6/9/21 2:05 PM, Rainer Deyke via Boost wrote:

...

On 08.06.21 09:33, Gavin Lambert via Boost wrote:

...

For this to work, one of the following must be true:    1. library B can figure out how program P chose to link to it and links to it the exact same way.

A header-only library cannot *link* to anything, or it is not header-only.

CMake defines targets for header-only libraries, which can have dependencies on other targets, including on static or shared libraries. So the program that "links" to the header-only library picks up its dependencies recursively. As a result, the linker is invoked with all (binary) libraries in the dependency tree.

If this functionality works the way I think does, then I think it should not be used.

And in what way do you think it works? (Honest question.)

...

Whether you can consider such library header-only in the first place is a philosophical question. I do, because otherwise any library that calls the standard library cannot be called header-only.

Actually, the standard library is a great example of what I am talking about.  A header-only library (B) should not care about the specific implementation of the standard library (A) being used by the program (P).  There is no link dependency from B to A.  There is a requirement from B to P that P must provide /a/ implementation of the C++ standard library, but B doesn't care if it is libstdc++ or libc++ or even a custom standard library implementation that is part of P and not a separate library at all.

That works for the standard-library, because you can be sure that one is available. (Otherwise your C++ compiler would be broken.) However, for other libraries A there is no way to guarantee that. Of course, you need to make sure that (any) dependency A is available on your build machine, otherwise building could never succeed. But the extra burden, to prepare the build-files (aka CMakeLists.txt) of your program P to find and explicitly configure even indirect dependencies (using `find_package`, `target_link_libraries`, `target_include_directories` etc.) is not to be underestimated. For a complex dependency-hierarchy this can become a nightmare, in particular if you need to find out in what order to process the individual build-files (`add_subdirectories`, `find_package` etc.). Instead, program P should only be expected to declare its direct dependencies (using `target_link_libraries`) and automatically get all the usage-requirements (aka dependencies) of these. This only requires, that every dependency (CMake target) carries its own usage-requirements. And, IMO, this should be true for header-only CMake targets, too.

Again, to make my point absolutely clear: header-only library B "using" library A does not imply a dependency from B to A. Instead, it implies a requirement on a program P that uses B to also link to A.  Header-only libraries use interfaces.  Programs link to implementations.

I just want to emphasize that even such interfaces are part of the usage-requirements I was talking about. Therefore B has a **usage-dependency** on A. And the CMake target for B should communicate this (through its usage-requirements) to its user, the program P (or more specific its CMake target). The usage-requirements of the CMake target for B will become the build-requirements of the CMake target for P (and could possibly even become its usage-requirements as well). And whether A is header-only or not is irrelevant. This (indirect) dependency from P to A can either mean linking against A or just having the correct include-search-path for the headers of A. The CMakeLists.txt file for building P needs to have this information either way. And this can be provided explicitly (as you prefer) which can become quite complex and tedious, or automatically and indirectly through B. Deniz PS: Maybe watching the first 11 minutes of my CMake talk [1] from Meeting C++ 2018 might help to better understand this target-centric approach CMake took. [1] https://youtu.be/y7ndUhdQuU8 -- BENOCS GmbH Dipl.-Inform. Deniz Bahadir Reuchlinstr. 10 D 10553 Berlin Germany Phone: +49 - 30 / 577 0004-22 Email: deniz.bahadir@benocs.com www.benocs.com Board of Management: Stephan Schroeder, Dr.-Ing. Ingmar Poese Commercial Register: Amtsgericht Bonn HRB 19378

Am 11.06.21 um 22:17 schrieb Rainer Deyke via Boost:

On 11.06.21 16:20, Deniz Bahadir via Boost wrote:

...

Am 11.06.21 um 11:17 schrieb Rainer Deyke via Boost:

...

If this functionality works the way I think does, then I think it should not be used.

And in what way do you think it works? (Honest question.)

I think it forces the users of library B to (indirectly) link to library A, as opposed any other library that implements the interface of A.  In the worst case, it might even unnecessarily force a specific version of A.

...

...

Actually, the standard library is a great example of what I am talking about.  A header-only library (B) should not care about the specific implementation of the standard library (A) being used by the program (P).  There is no link dependency from B to A.  There is a requirement from B to P that P must provide /a/ implementation of the C++ standard library, but B doesn't care if it is libstdc++ or libc++ or even a custom standard library implementation that is part of P and not a separate library at all.

That works for the standard-library, because you can be sure that one is available. (Otherwise your C++ compiler would be broken.) However, for other libraries A there is no way to guarantee that.

You guarantee it by documenting it as a requirement for the users of B.  The human users, not CMake.

...

Of course, you need to make sure that (any) dependency A is available on your build machine, otherwise building could never succeed. But the extra burden, to prepare the build-files (aka CMakeLists.txt) of your program P to find and explicitly configure even indirect dependencies (using `find_package`, `target_link_libraries`, `target_include_directories` etc.) is not to be underestimated. For a complex dependency-hierarchy this can become a nightmare, in particular if you need to find out in what order to process the individual build-files (`add_subdirectories`, `find_package` etc.).

In order to build P, you need to build the specific versions of all of the libraries that P requires, with the specific build settings that P requires.  The build process should never pick up random library versions found in system directories.  Program P must be able to say "use this modified version of libpng instead of the upstream version", "use LibreSSL instead of OpenSSL", and "don't use zlib at all, I'll provide my own version of the zlib functions".

You can even achieve that with CMake and its target-centric approach: If library B is a good citizen it uses `find_package` to look for its dependencies (more specifically, for the CMake target of its dependencies) and provides customization points (CMake options or simple variables). So if Program P wants to force a specific version of the indirect dependency A upon B it can set these customization points beforehand or it can just call `find_package` for that indirect dependency explicitly beforehand. The later call to `find_package` from B's CMakeLists.txt should then just return the same target (without even bothering searching). Of course, this requires careful crafting of B's CMakeLists.txt. (But it really can be worth it.) In case the indirect dependency A is header-only itself, it becomes even simpler because you do not have to cope with a library that might have been compiled using different/incompatible compiler-flags. IMO, CMake is quite powerful and one can achieve some really "funky stuff". But, as always, one should remember to not make it too complex or one will be unable to find anyone else who is willing to maintain it later. Deniz -- BENOCS GmbH Dipl.-Inform. Deniz Bahadir Reuchlinstr. 10 D 10553 Berlin Germany Phone: +49 - 30 / 577 0004-22 Email: deniz.bahadir@benocs.com www.benocs.com Board of Management: Stephan Schroeder, Dr.-Ing. Ingmar Poese Commercial Register: Amtsgericht Bonn HRB 19378

On 11/06/2021 9:17 pm, Rainer Deyke wrote:

Actually, the standard library is a great example of what I am talking about.  A header-only library (B) should not care about the specific implementation of the standard library (A) being used by the program (P).  There is no link dependency from B to A.  There is a requirement from B to P that P must provide /a/ implementation of the C++ standard library, but B doesn't care if it is libstdc++ or libc++ or even a custom standard library implementation that is part of P and not a separate library at all.

It does, however, need to be sure that it is using the same standard library as P (and with the same settings). Mixing multiple standard libraries into the same P will only end poorly -- which is usually also true (but less immediately obvious) when mixing multiple variants of other single libraries into P. You're assuming that by not specifying an explicit link from B to the standard library that it will inherently have the same settings as P. You're even mostly correct, because the standard library has a bit of special handling in the build tools. But this issue is not unique to the standard library, but applies to any dependency, including ones that you do have to explicitly tell the build tools about.

On 14/06/2021 5:34 pm, Rainer Deyke wrote:

...

You're assuming that by not specifying an explicit link from B to the standard library that it will inherently have the same settings as P. You're even mostly correct, because the standard library has a bit of special handling in the build tools.

No, I'm saying that by not having a separate compilation step, there is literally no way for B to have different settings than P, because there is nothing that could be compiled with different settings.  This is true regardless of the build system, because it is true at the compile level.  There is no set of arguments that the build system can pass to the compiler that would cause B to be built with different settings than P.

B is a header-only library.  To use B, the C++ preprocessor literally pastes the code from B into P.  Therefore, B /has/ no build settings. There are only the build settings from P, which are applied indiscriminately to P's own code and the code from B that P #includes.

That's actually completely false. The preprocessor itself *is* build settings -- defines can be defined differently (along with other compiler options) between the two builds. And yes, there may still be two builds. There is the build of P itself (which includes A) and the build of another library C that is linked into P but also includes A. Both of these can be via B or separately, and both can cause different code for B to be generated if the settings differ, which is an ODR violation. (Or cause two distinct copies of B to be generated, where C is a shared library with private visibility, for example.) For some examples, there's usually a compiler option to choose whether the standard library is linked as static or as shared, which in turn causes different #defines to be defined that will affect compilation. Another example is things that can vary due to the C++ language selected, or whether or not you've defined various "posix extension" flags, or library-specific configuration defines. Or something as simple as compiling one with exception support and one without. Having a header-only library is not a silver bullet to avoid these kinds of compilation incompatibilities. In fact it makes some of them worse.

On 08.06.21 01:58, Peter Dimov via Boost wrote:

Raffi Enficiaud wrote:

...

I have not looked at the recent developments for CMake in Boost. If the CMakeLists.txt I developed (under build/) is still there, it contained 2 targets: one for static and one for shared. Those were having different and explicit names.

I decided to not do this and instead use the "normal" CMake way of relying on BUILD_SHARED_LIBS.

It causes problems in the more complicated cases where a compiled Boost::X links to header-only Boost::Y which links to a compiled Boost::Z. Client code can choose to link explicitly to Boost::static_X, and if X used Z directly, it could link to Boost::static_Z, but it doesn't. And requiring all header-only libraries to also provide triplicate targets so that static/shared-ness can be propagated correctly is (I think) infeasible. So X links to Boost::Y, and Y has to choose whether to link to the static or shared Z, but it doesn't know how.

(b2 does this "right" by having static/shared as a build feature, and propagating it automatically. No need to split your targets by hand.)

Swimming upstream is sometimes justified when it results in something objectively better than the accepted CMake way, but I don't believe that's the case here, so BUILD_SHARED_LIBS will have to do. (Similarly, I tried to do something b2-like with the tests, but at the end reverting to the way CMake wants to do things proved better.)

Getting back to Boost.Test, I've duplicated the targets with an included_ prefix:

https://github.com/boostorg/test/commit/bce2d24c8b32f47f0403766fe4fee3e2e93a...

Thanks, it looks good to me and if it works for Edward, then all good :) BUILD_SHARED_LIBS is a global property inherited by all targets, with the drawback of not being able to explain all desirable configurations (# exponential in the number of targets). One example where it is not working: I need to run the unit tests of Boost.Test with shared, static and header only, so those 3 targets must exist. But this is not an isolated usage: as a project grow, it is sometimes needed that a specific lib exists in both shared and static variant. In any case, it is always possible (later, if needed) to define the shared/static explicitly and make an alias on the target that mimics BUILD_SHARED_LIBS. I have to thank Peter and all other boosters that participated to this effort. On a side note: I believe the "democratic" way of doing it does not work inside the current boost setup, and yield endless discussions not producing any artifacts. OTOH, the person doing that should have enough established authority & being respected within Boost for doing such a job. Peter has this, I do not. Coming back to Boost.Test, Edward + Peter: your call. Raffi

I know that in a project on which I am working I use Boost.Test in header-only mode and it works fine when using b2 to run tests for the project. I would like to think that if I try to use CMake to run tests for the project using the header-only variant of Boost.Test it will continue to work fine. That's all really.

Peter wrote that he implemented that in https://github.com/boostorg/test/commit/bce2d24c8b32f47f0403766fe4fee3e2e93a... So all you got to do is "link" to `Boost::included_unit_test_framework` and you can use it header-only. "Link" in CMake terms, no actual linking done, only declaring a dependency, and getting the headers into the include path. Although I would have used `header_only` as a suffix instead, but well, naming ;)

Andrey Semashev wrote:

...

Getting back to Boost.Test, I've duplicated the targets with an included_ prefix:

https://github.com/boostorg/test/commit/bce2d24c8b32f47f0403766fe4fee3e2e93a...

Not to start a bikeshedding argument, but maybe a more generic naming scheme would be preferable?

Since the normal unit_test_framework is used with #include and the "included" one is used with #include it seemed natural to call the target Boost::included_unit_test_framework in the second case. (Similarly with the execution monitors.)