[build] Tests automatically create header links, but library builds do not
If I remove my boost/ directory, and attempt to build, for example, atomic, by going into libs/atomic/build and invoking b2, it fails due to a missing header: D:\tmp2\libs\atomic\build>b2 toolset=msvc-8.0 ...found 1 target... ...updating 1 target... msvc.write-setup-script C:\Users\PETERD~1\AppData\Local\Temp\b2_msvc_8.0_vcvarsall_x86.cmd ...updated 1 target... ...found 1 target... ...updating 1 target... msvc.write-setup-script C:\Users\PETERD~1\AppData\Local\Temp\b2_msvc_8.0_vcvarsall_x86_amd64.cmd ...updated 1 target... Performing configuration checks - symlinks supported : no - junctions supported : yes - hardlinks supported : yes ...found 208 targets... ...updating 5 targets... common.mkdir ..\..\..\bin.v2\libs\atomic\build\msvc-8.0\debug\threading-multi compile-c-c++ ..\..\..\bin.v2\libs\atomic\build\msvc-8.0\debug\threading-multi\lockpool.obj lockpool.cpp ..\..\..\boost/atomic/detail/ops_msvc_x86.hpp(20) : fatal error C1083: Cannot open include file: 'boost/type_traits/make_signed.hpp': No such file or directory [...] Same if I do b2 --with-atomic from root. However, if I go into, say, libs/smart_ptr/test and invoke b2 there, it works: D:\tmp2\libs\smart_ptr\test>b2 toolset=msvc-8.0 ...found 1 target... ...updating 1 target... msvc.write-setup-script C:\Users\PETERD~1\AppData\Local\Temp\b2_msvc_8.0_vcvarsa ll_x86.cmd ...updated 1 target... ...found 1 target... ...updating 1 target... msvc.write-setup-script C:\Users\PETERD~1\AppData\Local\Temp\b2_msvc_8.0_vcvarsa ll_x86_amd64.cmd ...updated 1 target... Performing configuration checks - symlinks supported : no - junctions supported : yes - hardlinks supported : yes ...patience... ...found 1722 targets... ...updating 1253 targets... common.mkdir ..\..\..\bin.v2\libs\smart_ptr common.mkdir ..\..\..\bin.v2\libs\smart_ptr\test common.mkdir ..\..\..\bin.v2\libs\smart_ptr\test\yield_k_test~mt.test common.mkdir ..\..\..\bin.v2\libs\smart_ptr\test\yield_k_test~mt.test\msvc-8.0 common.mkdir ..\..\..\bin.v2\libs\smart_ptr\test\yield_k_test~mt.test\msvc-8.0\debug common.mkdir ..\..\..\bin.v2\libs\smart_ptr\test\yield_k_test~mt.test\msvc-8.0\debug\threading-multi mklink-or-dir ..\..\..\boost\smart_ptr Junction created for ..\..\..\boost\smart_ptr <<===>> ..\include\boost\smart_ptr link.hardlink ..\..\..\boost\predef.h Hardlink created for ..\..\..\boost\predef.h <<===>> ..\..\predef\include\boost\predef.h [...] Why is that? My first thought was that this was due to the fact that I was missing libs/Jamfile.v2 (I do various "clean" installations as an experiment to see what's needed and what isn't), but I put it back there and it doesn't help.
On Saturday 03 January 2015 19:00:30 Peter Dimov wrote:
If I remove my boost/ directory, and attempt to build, for example, atomic, by going into libs/atomic/build and invoking b2, it fails due to a missing header:
Why is that?
That's because Boost.Build is unable to parse #includes made with preprocessor macros. The TypeTraits headers are included from the implementation headers in Atomic, which in turn are included through a macro. Boost.Build does not find those implementation headers and hence do not make TypeTraits a dependency. We discussed this on the list in the MPL.Core-related thread.
Andrey Semashev wrote:
On Saturday 03 January 2015 19:00:30 Peter Dimov wrote:
If I remove my boost/ directory, and attempt to build, for example, atomic, by going into libs/atomic/build and invoking b2, it fails due to a missing header:
Why is that?
That's because Boost.Build is unable to parse #includes made with preprocessor macros. The TypeTraits headers are included from the implementation headers in Atomic, which in turn are included through a macro. Boost.Build does not find those implementation headers and hence do not make TypeTraits a dependency.
Yes, you're right, I picked 'atomic' at random, and it turned out to be exactly the wrong choice for this kind of test. Building 'system', for example, works. :-)
On 01/04/2015 03:58 AM, Peter Dimov wrote:
Andrey Semashev wrote:
On Saturday 03 January 2015 19:00:30 Peter Dimov wrote:
If I remove my boost/ directory, and attempt to build, for example, > atomic, by going into libs/atomic/build and invoking b2, it fails due to > a missing header:
Why is that?
That's because Boost.Build is unable to parse #includes made with preprocessor macros. The TypeTraits headers are included from the implementation headers in Atomic, which in turn are included through a macro. Boost.Build does not find those implementation headers and hence do not make TypeTraits a dependency.
Yes, you're right, I picked 'atomic' at random, and it turned out to be exactly the wrong choice for this kind of test. Building 'system', for example, works. :-)
Last time I recall you and Andrey had a discussion about the best way to handle this, and no agreement was reached? -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
On Monday 05 January 2015 11:20:06 Vladimir Prus wrote:
Last time I recall you and Andrey had a discussion about the best way to handle this, and no agreement was reached?
Yes, IIRC, Peter suggested to add fake #includes to the code and I didn't like it. I still think Boost.Build should link all headers before testing. In this case it should also be done before building a library. Is this possible? Alternatively, is it possible to add a dependency on a header-only library so that its include path (e.g. libs/type_traits/include) is added to compiler switches? This dependency would have to be transitive, so that if I add it to Atomic, other libraries that use Atomic also get the include path for TypeTraits. If this is done there would be no need for linking headers prior to building libraries.
On 01/05/2015 12:06 PM, Andrey Semashev wrote:
On Monday 05 January 2015 11:20:06 Vladimir Prus wrote:
Last time I recall you and Andrey had a discussion about the best way to handle this, and no agreement was reached?
Yes, IIRC, Peter suggested to add fake #includes to the code and I didn't like it. I still think Boost.Build should link all headers before testing. In this case it should also be done before building a library. Is this possible?
Your pull request does exactly that, so yes, it's possible. My original concern was that we don't understand why implicit-dependency is not working, now that we've established that it's not a Boost.Build bug, but rather MPL's use of preprocessor, I don't have an opinion either way, but I would much rather we implement a solution that everybody interested agrees with. I understand you prefer to just always link all headers. I'd be happy to apply that change if there are no -1 votes in this thread, and overall positive vote, by the time I'm back from public holidays next Monday. See http://www.apache.org/foundation/voting.html by what I mean by 'vote'. Does that work? -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com
On Monday 05 January 2015 12:29:06 Vladimir Prus wrote:
On 01/05/2015 12:06 PM, Andrey Semashev wrote:
On Monday 05 January 2015 11:20:06 Vladimir Prus wrote:
Last time I recall you and Andrey had a discussion about the best way to handle this, and no agreement was reached?
Yes, IIRC, Peter suggested to add fake #includes to the code and I didn't like it. I still think Boost.Build should link all headers before testing. In this case it should also be done before building a library. Is this possible?
Your pull request does exactly that, so yes, it's possible.
The pull request does this for testing, but not for building libraries. I understand we can similarly modify all libraries Jamfiles (and all tests and examples while we're at it), but I was more interested if it was possible to do it centrally, in Boost.Build.
My original concern was that we don't understand why implicit-dependency is not working, now that we've established that it's not a Boost.Build bug, but rather MPL's use of preprocessor, I don't have an opinion either way, but I would much rather we implement a solution that everybody interested agrees with.
I understand you prefer to just always link all headers. I'd be happy to apply that change if there are no -1 votes in this thread, and overall positive vote, by the time I'm back from public holidays next Monday. See http://www.apache.org/foundation/voting.html by what I mean by 'vote'. Does that work?
Works for me. Count me as +1. Happy holidays!
Vladimir Prus wrote:
I understand you prefer to just always link all headers. I'd be happy to apply that change if there are no -1 votes in this thread, and overall positive vote, by the time I'm back from public holidays next Monday. See http://www.apache.org/foundation/voting.html by what I mean by 'vote'. Does that work?
I'm at somewhere below -0.5 by that scale, "don't like it, no rational justification." My rational justification is: - The problem is caused by the inability of dependency scanners to track #includes via a macro; - This problem can be addressed by somehow informing the dependency scanner of the hidden dependency; - The only way that works for all such tools, Boost and non-Boost, is to add the hidden dependencies as #includes in an #if 0 section; - Therefore, if a library #includes via a macro, the proper thing to do, as a courtesy to dependency scanning tools, is to add such a section. Dependency scanners include bcp, boostdep, b2, and other build systems. Linking all headers does not address the underlying cause, hence my preference to not do it and fix the libraries instead.
Andrey Semashev wrote:
Yes, IIRC, Peter suggested to add fake #includes to the code and I didn't like it.
My suggestion still stands. We had this problem with Boost.Config, John added the includes in an #if 0 section instead of not liking it for weeks, and we haven't had a problem since. Sure, you could sweep the problem under the carpet by always generating all links, but the fact that any dependency scanners (such as boostdep and build systems) will not be aware of your dependencies will still remain.
On Mon, Jan 5, 2015 at 4:01 PM, Peter Dimov
Andrey Semashev wrote:
Yes, IIRC, Peter suggested to add fake #includes to the code and I didn't like it.
My suggestion still stands. We had this problem with Boost.Config, John added the includes in an #if 0 section instead of not liking it for weeks, and we haven't had a problem since.
Sure, you could sweep the problem under the carpet by always generating all links, but the fact that any dependency scanners (such as boostdep and build systems) will not be aware of your dependencies will still remain.
My opinion is that any tool that intends to parse C++ should do it right, Boost.Build included. Otherwise I cannot rely on the tool and, I don't want to mangle the code to make it happy. The situation is similar to Doxygen - it works most of the time, but it takes a lot of effort to make it work right with real code, and ironically this often makes the code less readable. There's another point: you'll have a problem when we want to differentiate dependencies based on some criteria (OS, compiler, C++ version, build configuraion, etc.) Code mangling doesn't help here, not until Boost.Build gets a proper C++ preprocessor. Actually I'd say that the ability of Boost.Build to parse C++ is a misfeature since a build system should typically be as language neutral as possible. I understand that Boost.Build is primarily targeted for Boost, which is C++, but still I think this is one point where it does more than it should.
Andrey Semashev wrote:
My opinion is that any tool that intends to parse C++ should do it right, Boost.Build included.
The tools do parse it right. It is not possible for a dependency scanner to figure out what macros are defined to what, in general - the predefined macros differ across compilers and systems, as do the explicit -D arguments. But even if the tool could figure it out somehow, it should not. That's because the configuration where the dependency scanner is being run may not match the configuration in which the dependency information will be used.
On Mon, Jan 5, 2015 at 7:40 PM, Peter Dimov
Andrey Semashev wrote:
My opinion is that any tool that intends to parse C++ should do it right, Boost.Build included.
The tools do parse it right.
It is not possible for a dependency scanner to figure out what macros are defined to what, in general - the predefined macros differ across compilers and systems, as do the explicit -D arguments.
But even if the tool could figure it out somehow, it should not.
That's because the configuration where the dependency scanner is being run may not match the configuration in which the dependency information will be used.
That doesn't match my definition of "right". :) Because it yields incorrect result with a valid C++ code. If it can't make a valid result, better not parse C++ at all and declare a new protocol for dependency definition.
Andrey Semashev wrote:
That doesn't match my definition of "right". :) Because it yields incorrect result with a valid C++ code. If it can't make a valid result, better not parse C++ at all and declare a new protocol for dependency definition.
The protocol for dependency definition is already defined. It consists of lines of the form #include <header> or #include "header" inline in the C++ source files. You just refuse to follow it. Any other protocol will be equivalent to this one, in that it necessarily would need to contain the same information.
On Mon, Jan 5, 2015 at 11:05 AM, Peter Dimov
Andrey Semashev wrote:
That doesn't match my definition of "right". :) Because it yields
incorrect result with a valid C++ code. If it can't make a valid result, better not parse C++ at all and declare a new protocol for dependency definition.
The protocol for dependency definition is already defined. It consists of lines of the form
#include <header>
or
#include "header"
inline in the C++ source files.
I should point out that is the protocol for C, C++, Objective C, and Objective C++. All of which are supported by Boost Build.
You just refuse to follow it.
Any other protocol will be equivalent to this one, in that it necessarily would need to contain the same information.
+1 -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
Andrey Semashev wrote:
Alternatively, is it possible to add a dependency on a header-only library so that its include path (e.g. libs/type_traits/include) is added to compiler switches? This dependency would have to be transitive, so that if I add it to Atomic, other libraries that use Atomic also get the include path for TypeTraits. If this is done there would be no need for linking headers prior to building libraries.
We touched on this earlier. Yes, I believe it's possible, via usage-requirements. But this scheme, if applied globally, would require all libraries (including header-only ones) to explicitly list their dependencies in their Jamfiles. It will also result in some rather long command lines for the libraries at the higher levels. I suppose that what you have in mind is to only apply this to specific libraries, such as Atomic or mpl-core. But it will also require applying it to their dependencies, such as TypeTraits, otherwise headers indirectly included from Atomic via TypeTraits will (potentially) not be linked and found.
On Mon, Jan 5, 2015 at 4:16 PM, Peter Dimov
Andrey Semashev wrote:
Alternatively, is it possible to add a dependency on a header-only library so that its include path (e.g. libs/type_traits/include) is added to compiler switches? This dependency would have to be transitive, so that if I add it to Atomic, other libraries that use Atomic also get the include path for TypeTraits. If this is done there would be no need for linking headers prior to building libraries.
We touched on this earlier. Yes, I believe it's possible, via usage-requirements. But this scheme, if applied globally, would require all libraries (including header-only ones) to explicitly list their dependencies in their Jamfiles. It will also result in some rather long command lines for the libraries at the higher levels.
I suppose that what you have in mind is to only apply this to specific libraries, such as Atomic or mpl-core. But it will also require applying it to their dependencies, such as TypeTraits, otherwise headers indirectly included from Atomic via TypeTraits will (potentially) not be linked and found.
The idea is somewhat connected to the recent topic about the directory structure. If we intend to remove the $BOOST_ROOT/boost (or $BOOST_ROOT/include/boost) directory in the developer setups then we'd have to make a change like this to all Jamfiles. But for now, as long as we have that common directory, such massive changes are not necessary, and we could get away with a less intrusive change - just to automatically invoke the headers target before building anything. I was hoping that such a change could be done solely within Boost.Build. Of course, if that's not possible then we could discuss those larger changes.
participants (4)
-
Andrey Semashev -
Peter Dimov -
Rene Rivera -
Vladimir Prus