Andrey Semashev wrote:

Note that this does not mean that there will be no such common directory in users' setups. The equivalent of 'make install' or package installation can (and I'm sure will, at least for some time) form the boost directory with all public headers. The location of this directory is system-specific.

Happy New Year :-) Let's assume Windows then. Let's say that the package installer installs libs/this and libs/that into C:\boost-1.57.0, and then does the equivalent of "b2 headers". The header links go into C:\boost-1.57.0\boost, and the user needs to add C:\boost-1.57.0 to the include path. I say that -I C:\boost-1.57.0\include is better and more consistent with expectations.

On Thu, Jan 1, 2015 at 10:02 AM, Edward Diener wrote:

On 1/1/2015 9:50 AM, Peter Dimov wrote:

...

Andrey Semashev wrote:

...

Note that this does not mean that there will be no such common directory in users' setups. The equivalent of 'make install' or package installation can (and I'm sure will, at least for some time) form the boost directory with all public headers. The location of this directory is system-specific.

Happy New Year :-)

Let's assume Windows then. Let's say that the package installer installs libs/this and libs/that into C:\boost-1.57.0, and then does the equivalent of "b2 headers". The header links go into C:\boost-1.57.0\boost, and the user needs to add C:\boost-1.57.0 to the include path. I say that -I C:\boost-1.57.0\include is better and more consistent with expectations.

+1

Normally software distributions have their header files off some directory, usually called "include", from their product's install directory. Boost having their header files off of the install directory is an anomaly and I concur with Peter that this should be changed for better consistency with other software.

I believe we already do that with "b2 install" at the Boost root. As can be seen from the Jamroot docs.. < https://github.com/boostorg/boost/blob/master/Jamroot#L31>. Which are listedn on "b2 --help" at the Boost root. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

On 02.01.2015 06:35, Peter Dimov wrote:

Redownloading the whole package seems a but redundant, doesn't it? Would it not make sense to leave it all in c:\boost-1.57.0 and just build things there as needed? Am I really the only one who does that?

FWIW, as a library user, that's what I've done all along. Download, unpack, build --with-something, add somewhere\boost\ to include dirs and \somewhere\boost\stage\lib to lib dirs. Never used "install" or "headers" option. - Asbjørn

On Fri, Jan 2, 2015 at 9:13 AM, Daniela Engert wrote:

Am 02.01.2015 um 08:36 schrieb Asbjørn:

...

On 02.01.2015 06:35, Peter Dimov wrote:

...

Would it not make sense to leave it all in c:\boost-1.57.0 and just build things there as needed? Am I really the only one who does that?

FWIW, as a library user, that's what I've done all along. So do I.

FWIW.. *nix distributions, and I suspect most *nix users, don't do that. They do a binary packages install from whatever the packager of the system gives them. Which is almost certainly the result of a packager running "b2 install". And hence they never see the original & full Boost source tree. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

On 01. jan. 2015 17:02, Edward Diener wrote:

On 1/1/2015 9:50 AM, Peter Dimov wrote:

...

Andrey Semashev wrote:

...

Note that this does not mean that there will be no such common directory in users' setups. The equivalent of 'make install' or package installation can (and I'm sure will, at least for some time) form the boost directory with all public headers. The location of this directory is system-specific.

Happy New Year :-)

Let's assume Windows then. Let's say that the package installer installs libs/this and libs/that into C:\boost-1.57.0, and then does the equivalent of "b2 headers". The header links go into C:\boost-1.57.0\boost, and the user needs to add C:\boost-1.57.0 to the include path. I say that -I C:\boost-1.57.0\include is better and more consistent with expectations.

+1

Normally software distributions have their header files off some directory, usually called "include", from their product's install directory. Boost having their header files off of the install directory is an anomaly and I concur with Peter that this should be changed for better consistency with other software.

I think I am missing the point. Unless libs is renamed src, which I dislike, the purpose really escapes me. What is the end goal? As it is, boost headers can be installed in a folder called include by installers on any platform, there is nothing that prevents that as things are now, and in fact it is done by several. It is up to whomever make the packages for the deployment. I am more in line with Andrey Semashev here in that the goal should be to fix dependencies between modules to the level we can conveniently get rid of b2 headers all together. To build and test libraries we should rather use -I commandline directives for each dependency. To emulating a deployment structure it make more sense to have a build target like the current stage target do that in a staging area. b2 already do that for libraries, so why not the includes as well. Within each module there is already an include, src, test, build directory structure. The b2 headers staging of headers to the boost folder was just to ease the transfer to the git modules and is a temporary solution -- hopefully. -- Bjørn

On 04. jan. 2015 16:48, Peter Dimov wrote:

Bjørn Roald wrote:

...

What is the end goal?

That was, and remains, a very good question.

The end goal is a tool 'bpm' that can do this:

D:\tmp3>bpm install filesystem bpm: installing module 'filesystem' bpm: installing module 'assert' bpm: installing module 'config' bpm: installing module 'core' bpm: installing module 'detail' bpm: installing module 'functional' bpm: installing module 'io' bpm: installing module 'iterator' bpm: installing module 'range' bpm: installing module 'smart_ptr' bpm: installing module 'static_assert' bpm: installing module 'system' bpm: installing module 'type_traits' bpm: installing module 'mpl' bpm: installing module 'preprocessor' bpm: installing module 'function' bpm: installing module 'function_types' bpm: installing module 'integer' bpm: installing module 'optional' bpm: installing module 'typeof' bpm: installing module 'utility' bpm: installing module 'concept_check' bpm: installing module 'conversion' bpm: installing module 'tuple' bpm: installing module 'array' bpm: installing module 'regex' bpm: installing module 'align' bpm: installing module 'predef' bpm: installing module 'throw_exception' bpm: installing module 'bind' bpm: installing module 'move' bpm: installing module 'build' bpm: the following libraries need to be built: filesystem regex system bpm: (use ./b2 to build) bpm: recreating header links

which should be self-explanatory.

When it says "installing module 'foo'", it means "downloading libs/foo" (except for build, where it downloads tools/build and a few other files.)

I like the intentions here. I think having a platform agnostic tool for source level packet management is a good way to go. I would be more skeptical if you intended make the next generic packet manager from scratch for binary distribution. It is just too complex, and there are too much competition too have a reasonable chance of getting wide adaption. That said, useful meta data from boosdep, bpm, b2, etc., with some HOWTOs, could make life better for boost packet maintainers trying to make good binary distributions of boost for various platforms. A few opinions: A source packet manager, like bpm, should copy headers from the installed modules' into the $BOOST/include/boost directory. The bpm tool should not depend on headers in $BOOST/boost, nor depend on ./b2 for staging headers in $BOOST/boost or $BOOST/include/boost. Further bpm should avoid any symbolic link, junction, or hard link trickery like "b2 headers" attempts, it is just too tricky to do right for everyone, in particular automated symbolic link creation on Windows is hard and just not worth it. We are talking about an installation, so keep it simple. For the b2 build I would suggest changing jam files so b2 is specifying a -I for each module the build depend on, rather than a single -I $BOOST/include to where bpm staged the headers. That will keep the two tools independent of each other. I would also like to suggest having bpm install all modules, including build in a "modules" folder together with a custom bootsstrap and set of Boost.Build project root files, and stage libraries in a "lib" folder rather than stage/lib. -- Bjørn

On 06. jan. 2015 01:49, Peter Dimov wrote:

Bjørn Roald wrote:

...

I like the intentions here. I think having a platform agnostic tool for source level packet management is a good way to go. I would be more skeptical if you intended make the next generic packet manager from scratch for binary distribution.

Supporting binary distributions is actually a very interesting idea. It turns out that the only thing that prevents the current bpm to be able to install a binary distribution is that it requires that a module is completely contained into a single directory (libs/$module), so it will not permit the downloaded archive to extract stage/lib/libboost_$module_whatever.

I could however package the pre-built binaries in libs/$module/stage if I had the inclination, and then make bpm symlink stage/lib/libboost_$module_whatever to libs/$module/stage/lib/libboost_$module_whatever in the exact same manner as headers are handled.

Food for future thought.

I am not sure I follow all, but if some future version of bpm support pre-built packages then a new user will likely expect bpm install filesystem to install only the include and lib folders with binaries for the default toolchain on the download host, as this is what most users need, no more, while bpm install filesystem-doc filesystem-src filesystem-examples would install the respective optional extras, Hopefully globing or regexp would work so: bpm install filesystem* would instal the lot. I guess it can be turned around, so: bpm install filesystem-bin or something like bpm install filesystem-clang-3.5 for clang 3.5 binaries on standard host ABI I think you may want to give the tools command line convention some thought so it retain a growth potential for future support for handling of binary packages.

...

A source packet manager, like bpm, should copy headers from the installed modules' into the $BOOST/include/boost directory. The bpm tool should not depend on headers in $BOOST/boost, nor depend on ./b2 for staging headers in $BOOST/boost or $BOOST/include/boost.

That is what it does, yes.

Are you sure you _copy_ files, I think you say something else further down ;-)

...

Further bpm should avoid any symbolic link, junction, or hard link trickery like "b2 headers" attempts, it is just too tricky to do right for everyone, in particular automated symbolic link creation on Windows is hard and just not worth it. We are talking about an installation, so keep it simple.

Well... I already wrote the code to symlink/junction/hardlink things, so unless it turns out to create more problems than it solves, I'd rather keep it. :-)

OK, good luck with that.

...

For the b2 build I would suggest changing jam files so b2 is specifying a -I for each module the build depend on, rather than a single -I $BOOST/include to where bpm staged the headers.

The problem here, as I already mentioned in the other thread, is that the library Jamfiles will need to be maintained with the correct dependency information so that the usage-requirements correctly add the required include directories.

I don't think the maintenance of such dependencies in build files are hard, test build would cause errors on missing dependencies. b2 could help with dependency analysis and warn on problems and suggest changes. Generally it will also raise the level of caution with regard to dependency issues, as you actually see somewhere an explicit mention in your module og the other modules you depend directly on. #include <...> are not quite the same as it is not always clear you add an extra dependency. But I guess that is just an opinion. The bigger question is if time is right for this, it is an interruptive thing and a fair bit of jam files need to be modified. I would think unless it would actually help efforts of reducing inter boost dependencies, it is better to wait until those dependencies are reduced as far as it goes, then this can be reconsidered.

I am not saying that this will not be a good thing to do, at some unspecified future point. But I wanted to get things working with minimal changes to the status quo.

100% understood

...

I would also like to suggest having bpm install all modules, including build in a "modules" folder together with a custom bootsstrap and set of Boost.Build project root files, and stage libraries in a "lib" folder rather than stage/lib.

It would've been nice for the current structure to have all modules in modules/ or components/, rather than some in libs/ and some in tools/. As it is, I have a special case for the 'build' module to be in tools/build.

Fair enough, but it sort of escape me why you are so determined to get the headers into a conventional "include" directory, on the grounds that that is what users expect, while you seem perfectly fine with keeping the libraries hidden inside a very unconventional "stage" directory. What other project or software distribution use a "stage" directory? I can not think of one.

But as I said, I wanted bpm to work on the current structure.

Why? It is not like that structure exist before bpm download files. So there is no current structure to work on. Am I missing something? -- Bjørn

On 06. jan. 2015 03:42, Peter Dimov wrote:

Bjørn Roald wrote:

...

Fair enough, but it sort of escape me why you are so determined to get the headers into a conventional "include" directory, on the grounds that that is what users expect, while you seem perfectly fine with keeping the libraries hidden inside a very unconventional "stage" directory. What other project or software distribution use a "stage" directory? I can not think of one.

There are two reasons; the first one is that lib and libs are too close and will create a lot of confusion;

Which would be solved if there where no libs directory and all its subdirectories was in the modules directory

the second one is that I'm not very proficient with Boost.Build, so I kept my changes to a minimum.

I am not a sharp Boost.Build knife myself, so I am not sure how much this involves. But I think it may be relatively trivial, I may spend some time this week end looking at it.

...

...

But as I said, I wanted bpm to work on the current structure.

Why? It is not like that structure exist before bpm download files. So there is no current structure to work on. Am I missing something?

bpm itself wouldn't care, but if I moved libs/ to modules/ or components/ and stage/lib/ to lib/, I would have needed to patch all references to libs in Boost.Build (and potentially anywhere else), and I haven't investigated whether this is feasible.

I would expect all those references are to into the bin.v2 file structure, not to stage.

Perhaps it is, I just didn't check. (I would have then needed to maintain these fixes, as well.)

I am not sure there is anything that need to be changed for this, I will test when I gent some spare time. -- Bjørn

On Thursday 01 January 2015 16:50:22 Peter Dimov wrote:

Andrey Semashev wrote:

...

Note that this does not mean that there will be no such common directory in users' setups. The equivalent of 'make install' or package installation can (and I'm sure will, at least for some time) form the boost directory with all public headers. The location of this directory is system-specific.

Happy New Year :-)

You, and the list, too. :)

Let's assume Windows then. Let's say that the package installer installs libs/this and libs/that into C:\boost-1.57.0, and then does the equivalent of "b2 headers". The header links go into C:\boost-1.57.0\boost, and the user needs to add C:\boost-1.57.0 to the include path. I say that -I C:\boost-1.57.0\include is better and more consistent with expectations.

Yes, I'm ok with that. Windows is a white crow, as usual, so we need to come up with some solution on our part.

On 01. jan. 2015 22:02, Peter Dimov wrote:

...

I say that -I C:\boost-1.57.0\include is better and more consistent with expectations.

In addition, it also makes more sense even on its own. It's more consistent for "b2 headers" to create a "virtual" directory $BOOST/include from the contents of $BOOST/libs/*/include;

"b2 headers" should go away, it should not be needed any more when dependencies are cleaned up to a reasonable level. However, "b2 stage" could by default have same effect as "b2 install --prefix=.", then you will get exactly what you want I think.

putting the contents of $BOOST/libs/*/include into $BOOST doesn't seem right.

agree, the headers does not need to go anywhere before "install" or "stage" targets are called explicitly. But to get there the build rules for boost libraries and tests need -I../../system/include -I../../core/include ... etc. on command line rather than -IC:\boost_1.56.0

This would have been obvious if we had headers in libs/*/include, but since these directories always have their headers in a boost/ subdirectory, "b2 headers" puts nothing directly into $BOOST, so it works fine. Still, I can't think of any reason of insisting upon the current structure.

agreed

Yes, "b2 install" will put things in their proper place (on Linux), and yes, even without "b2 install" I can easily symlink /usr/include/boost to $BOOST/boost (on Linux), so it doesn't actually hurt (on Linux).

On Windows, one typical use is to unpack the Boost distribution into its final location and just b2 stage (without b2 install). Which works fine, and b2 then says, please add $BOOST to your include path and $BOOST/stage/lib to your library path. Which also works fine, except that first thing should, in my opinion, be $BOOST/include.

Why is there a reason this hurts more on Windows than on Linux?

What are the benefits of not putting the headers into include/? None as far as I can see.

Just avoiding the pain of a needless change if "b2 headers" is going away anyway, To be consistent you also need to change stagedir to $BOOST rather than $BOOST/stage or put headers in $BOOST/stage/include so "b2 stage" by default produce $BOOST/include $BOOST/lib or $BOOST/stage/include $BOOST/stage/lib -- Bjørn

On 02. jan. 2015 06:16, Peter Dimov wrote:

Bjørn Roald wrote:

...

However, "b2 stage" could by default have same effect as "b2 install --prefix=.", then you will get exactly what you want I think.

No, this is not what I want. I don't want two copies of the headers, one in include/boost/, and one in boost/ (that would be pretty stupid). I don't want a symlink include/boost -> boost (somewhat less stupid).

You cut out too much of my post to see the context here. My point was that when the $BOOST/boost folder is gone, because the "b2 headers" target is gone, there will not be two copies, only one. And there will not be symbolic links or hard links, it will be real copies of every file that is copied by the install or stage rules. Note that these files are staged like this as convenience for the end user which is using boost, not for building or testing boost libraries.

When I go somewhere inside the Boost source tree, and use b2 to build a target, and that target has #include , I want b2 to create a link include/boost/shared_ptr.hpp -> libs/smart_ptr/include/boost/shared_ptr.hpp, and then to use <include>include, rather than creating a link boost/shared_ptr.hpp -> libs/smart_ptr/include/boost/shared_ptr.hpp and using <include>. .

Note that this has nothing to do with staging or installing. The "headers" target is an implicit dependency. Even if I had a copy of the headers in include/boost/, b2 would still recreate them in boost/ each time I build something.

Not if the "b2 headers" target is removed all together.

I agree that this would not be an issue if b2 used separate include paths for each library.

Right, so if that is the direction things are going, then the "b2 headers" target will be redundant and the task of staging header files for users belongs naturally in the install and stage rules. If this is going to happen any time the next few years, then I think moving "b2 headers" target directory into $BOOST/include now is just needless noise regardless of its potential merit of a $BOOST/include with regard to user expectations. -- Bjørn

Vladimir Prus wrote:

The only important aspect though, speaking as release manager, is that if we change this now, we'll likely break a huge number of user build and automation scripts that expect current layout, lots of instructions on the web using current layout, and so forth.

Not if we symlink boost to include/boost, I think.

I agree with Andrey about transitioning Boost itself to using -I's instead of referring to $BOOST_ROOT/boost. It would save me much heartache as the headers target causes issues when your own project uses Boost-Build and refers to the Boost Libraries. On 1/3/15, 1:12 AM, "Vladimir Prus" wrote:

The only important aspect though, speaking as release manager, is that if we change this now, we'll likely break a huge number of user build and automation scripts that expect current layout, lots of instructions on the web using current layout, and so forth. That's easily person-months of pain. Do we want it so badly?

I would believe that most people use the zip archives for automation or a specific commit. I would propose a header only zip for those who do not wish to run the build. If they are tracking the master/develop branch they should expect breaking changes to occur once in a while. The headers target broke my builds when it was introduced. - Thomas

Beman Dawes wrote:

Or maybe there will be a new "packaged-boost" top level super-project. But that's not clear yet.

This is a bit too complex for my taste; we're supposed to be removing complexity here, not adding it. :-) The perfect scenario in my mind is that, to get a modular release, one checkouts the appropriate revision of the source tree and runs "package.bat" or "package.sh" at the root. (Everyone should be able to do that, not just a Boost release manager; people should be able to prepare a modular boost repository for their team or organization and hand their co-programmers a copy of bpm that installs from it.) Maintaining a shadow super-project doesn't fit into this picture.

On Tue, Jan 6, 2015 at 10:37 AM, Peter Dimov wrote:

Beman Dawes wrote: ...

...

* The directory structure installed by a Boost Package Manager, i.e. bpm. The introduction of a new modular way to install Boost is the ideal time to introduce a new install directory structure. Now. Not later. Don't release bpm for production use until we are happy with the directory structure it will install.

The problem here is obviously that the version of Boost.Build installed by bpm would need to support the new directory structure. When the two structures (source tree and bpm) are the same, or supported by the same Boost.Build, the package script can just "tar cf build.tar.lzma tools/build/ Jamroot boostcpp.jam libraries/Jamfile.v2" from the source tree, and call it a day. But if the bpm structure requires a custom version of Boost.Build, (a) the package script would have to do more work and (b) this custom version of Boost.Build will need to be tested separately. In the same structure case, the packager may (reasonably) assume that Boost.Build has already been tested.

Understood. I'm hoping that Boost.Build itself is the same for either distribution directory structure, and that the only thing that differs <boost-root>/Jamroot and perhaps some of the other <boost-root> scripts. Maybe the differing files could like in a different boostorg repo or a different boostorg/boost branch.

PS: Beman, you have an open pull request in system for meta/libraries.json.

Grrr... I need a little app that will look at GitHub on some schedule and tell me if there are any outstanding pull requests in any of my repos. Thanks, --Beman

Beman Dawes wrote:

I'm hoping that Boost.Build itself is the same for either distribution directory structure, and that the only thing that differs <boost-root>/Jamroot and perhaps some of the other <boost-root> scripts.

It would be nice if we could have a file that defines path constants (say) BOOST_INCLUDE BOOST_LIBS BOOST_STAGE so that it was the only place that would need to be patched. All explicit path references throughout the Jamfiles need to be changed to refer to these though. One could then have, for example, BOOST_INCLUDE = $(BOOST_ROOT)/include BOOST_LIBS = $(BOOST_ROOT)/modules BOOST_STAGE = $(BOOST_ROOT) If Boost.Build itself could then be moved from tools/build to modules/build, all will be well.

Paul A. Bristow wrote:

And that it [bpm] really does work nicely on all the platforms too, not just Linux.

I think that Linux may actually be the platform with least demand for a Boost package manager, as all distributions already have their own package managers and provide packages for Boost through them.