Stefan Seefeld wrote:

On 2014-05-29 16:22, Julian Gonggrijp wrote:

...

1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.

I very much applaud all efforts on 1., but I'm slightly against 2. insofar as it reduces the motivation to work on 1.

(Read: If by 2. you mean the tracking of dependencies, I think this is very useful in support of the overall goal. In contrast, if you mean tools to clone files automatically, that would provide incentives to continue carelessly adding unneeded dependencies, so I'd vote against it.)

By point 2. I meant a script that looks up what other submodules a particular Boost submodule depends on, and checks out those submodules in order to provide the user with a local copy (which can then be installed). I believe this would be a disincentive for library authors to add more dependencies, because library interdependencies become more marked in this way (currently the dependencies are invisible because users always download all of Boost). Nat Goodspeed suggested a similar positive effect for point 2, where the existence of a file that documents the dependencies of a library may motivate people to reduce the number of dependencies. Would you vote in favour of that? Either way, I'll count your vote in favour of point 1.

Good luck !

Thank you, and thanks to the others who voted so far. -Julian

Glen Fernandes wrote:

On Thu, May 29, 2014 at 1:22 PM, Julian Gonggrijp wrote:

...

- Manually: this is cumbersome and generally hard because there are many (undocumented) interdependencies between Boost libraries. - Automatically: as far as I know Ryppl is the only existing attempt at providing automated dependency handling for Boost, but it is a very ambitious project that attempts to do much more and it is currently neither finished nor being actively developed.

I think there could be a less ambitious but faster way to provide automated dependency handling. It would take no more than some static configuration, to list for each library what other Boost libraries it depends on, combined with a simple script that automatically traverses the dependency graph and clones just the necessary submodules. This script may then be hooked into the bootstrap script. I think I could write the dependency handler, and I would be willing to, though I can't currently provide any indication of when I would be able to start.

It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).

In my reply to Robert Ramey just now I mentioned the possibility to use Boost.Bjam's <toolset> to handle this. There might be other ways as well. Given that it is probably feasible to account for this, would you vote in favour of what I propose?

For example:

Boost.Align should only be considered as depending on Boost.TypeTraits (for boost::alignment_of) if BOOST_NO_CXX11_HDR_TYPE_TRAITS is defined; otherwise it uses the C++11 standard library (for std::alignment_of). Similarly, it only depends on Boost.Utility (for boost::addressof) if BOOST_NO_CXX11_STD_ADDRESSOF is defined; otherwise it uses the C++11 standard library (for std::addressof).

For anyone wanting to consume Boost.Align, should they have a conforming C++11 implementation, they should see Boost.Align's dependencies as only Boost.Config and Boost.Integer. Otherwise, someone consuming Boost.Align with a totally non-conforming C++11 implementation should see its dependencies as: Boost.Config, Boost.Integer, Boost.Utility, Boost.TypeTraits.

Glen

Glen Fernandes wrote:

It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).

It would be nice, but is it implementable? How could the dependency tracker determine what Boost.Config will define?

Nat Goodspeed wrote:

On Fri, May 30, 2014 at 8:31 AM, Peter Dimov wrote:

...

Glen Fernandes wrote:

...

...

It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).

...

It would be nice, but is it implementable? How could the dependency tracker determine what Boost.Config will define?

I've been assuming that Julian's proposal will require, at least initially, that someone (the library maintainer, an interested volunteer) manually edit a specification in some syntax of the other libraries on which this one directly depends. That information could be informed by, but need not be identical to, the output of a tool.

In that case, Glen's suggestion only requires that the specification syntax include some way of annotating each dependency: "only for the following toolsets..."

It may also be possible to write a (wave-based?) tool that analyzes the preprocessor conditions under which each #include is reached, but I see that as very much a separable (and deferrable) effort.

The initial topic of this thread was 'Does Boost have modularization as a goal'. There seems to have been topic-creep. Nevertheless, I suggest you and anyone else interested read http://www.cmake.org/cmake/help/git-master/manual/cmake-compile-features.7.h... which I implemented, and which allows transitive propagation of compile feature requirements at a buildsystem level. Especially if you think 'Does this library use C++11' and 'Does this library use C++14' are good questions, you should reconsider and re-frame your question in terms of what features are required from the compiler instead. Especially consider that in the light of what Robert Ramey wrote about 4 variants of C++. There are not 4 variants. There are many more than 4. There are as many variants as there are versions of compilers you support, which each support different features. No MSVC release 'supports C++11' (or C++98, if you want to be very pedantic), but it does support some features described in it which other compilers also support. That is what is relevant, not the C++ standard version: http://www.kdab.com/modern-cmake-with-qt-and-boost/#compile-feature-specific... CMake is aware of all C++11 and C++14 features and can depend on a target or not depending on whether the compiler in use supports a particular feature or features: http://www.cmake.org/cmake/help/git-master/prop_gbl/CMAKE_CXX_KNOWN_FEATURES... though someone would have to contribute and maintain a listing for MSVC supported features something like: https://gitorious.org/cmake/steveires-cmake/source/0156b7f4:Modules/Compiler... Even if you don't rely on CMake, you should consider a similar design with b2 or any new tool you create. Trying to keep the information in Boost.Config and extract it for use by the buildsystem or dependency resolver would be awkward. You can move the information about what compiler has what feature into b2 and use that information to generate Boost.Config, as CMake does: http://www.cmake.org/cmake/help/git-master/module/WriteCompilerDetectionHead... HOWEVER: All of this is off-topic for this thread. I'm listing it only because people are thinking too far down the line and getting stuck. I recommend dropping discussion of tooling and how to record/resolve dependencies. Understand that there are solutions for what will come up down the line and you don't need to worry about them now. I recommend you return to the question of whether Boost wants to modularize or not. If you do want that, then consider whether my recommendations linked in the start of the thread are specific or concrete. There were suggestions in this thread that my recommendations were not concrete or specific or didn't show any benefit. I don't understand or agree with those conclusions, but I think those are the useful and small questions you guys should focus on if you want to make any conclusive progress at all on this issue. Thanks, Steve.

Op 30 mei 2014 om 16:33 heeft "Peter Dimov" het volgende geschreven:

Stephen Kelly wrote:

...

I recommend you return to the question of whether Boost wants to modularize or not.

The root of the problem is that absence of modularity is (a) invisible, and (b) carries no cost.

It is invisible because there is no current easy way for developers and users to see a dependency report.

100% agree here.

It carries no cost because currently Boost developers pay no price at all for introducing a dependency and derive no benefit from severing one. (Realistically, neither do most users.)

Everyone except for the maintainer pays a price for additional dependencies. Hence everyone except for the maintainer also benefits from fewer dependencies. This is an issue that has been raised before. I do agree that having visible dependencies would increase the incentive for maintainers to reduce those dependencies. Making the dependencies visible is an intended side effect of part 2 of my plan. -Julian

On Fri, May 30, 2014 at 11:03 AM, Nat Goodspeed wrote:

Steven, you specifically mean this post, yes?

Apologies, Stephen! My eye obviously caught on your e-mail address rather than your real name. :-(

Nat Goodspeed wrote:

...

If you do want that, then consider whether my recommendations linked in the start of the thread are specific or concrete.

Steven, you specifically mean this post, yes? http://lists.boost.org/Archives/boost/2013/10/207384.php

I posted a follow-up too: http://thread.gmane.org/gmane.comp.lib.boost.devel/245078/focus=245806 Many respondents in this thread would benefit from reading the links Julian posted I think. Many have not done so it seems.

Stephen Kelly wrote:

[...]

HOWEVER: All of this is off-topic for this thread. I'm listing it only because people are thinking too far down the line and getting stuck. I recommend dropping discussion of tooling and how to record/resolve dependencies. Understand that there are solutions for what will come up down the line and you don't need to worry about them now.

I recommend you return to the question of whether Boost wants to modularize or not.

I agree with this point. Thank you for this remark, Stephen.

If you do want that, then consider whether my recommendations linked in the start of the thread are specific or concrete. There were suggestions in this thread that my recommendations were not concrete or specific or didn't show any benefit.

I don't understand or agree with those conclusions, but I think those are the useful and small questions you guys should focus on if you want to make any conclusive progress at all on this issue.

For these reasons, manual specification of dependencies can be reliable only if these specifications are tested as part of the regression tests. That is, if the tests for module X operate in an environment in which only the stated (and transitive) dependencies of X are present.

This looked impractical to me at first, but on second thought, it probably can be done with just a modification to "b2 headers" - there's no need to actually delete the libraries not listed as dependencies, just their header links.

Yes! The regression tests should definitely use the specification to build/test the library.

This might make the full regression run take longer (if there are libs that have dependencies on non-header-only libs, they will end up getting built more than once). On the other hand, once the regression tests get setup to test modular libraries, we can break up the monolitic regression runs into much more manageable pieces (each library + dependencies)...each of which will run much faster than the whole.

That would certainly be great, but what are we tracking as dependencies here - the headers? Or everything needed to test the lib? The latter is likely to be a much bigger list. John.

Peter Dimov wrote:

Glen Fernandes wrote:

...

It would be nice if both approaches (static configuration, automatic identification) take into account - or allow you to specify - conditional dependencies (e.g. dependencies based on identification of supported or unsupported features determined by Boost.Config).

It would be nice, but is it implementable? How could the dependency tracker determine what Boost.Config will define?

Tool detection using the <toolset> from Boost.Jam, perhaps combined with some manual configuration, although it seems that Peter Dimov's dependency traker might be sufficient. -Julian

John Maddock wrote:

We also need sensible policies for dealing with optional components - a good example would be libraries that provide serialization support in a separate optional header. The library as such does not require Boost.Serialization, but quite rightly the optional "bridging" support is there. I asked about this last time this topic came up, but I saw no good answer?

I'm not sure whether this is actually a significant problem in practice. An automated dependency tracker will be confused, but that would simply be a matter of marking up the bridge header as "do-not-track" in some way, wouldn't it?

On 05/30/2014 01:17 PM, John Maddock wrote:

On 30/05/2014 10:44, Andrey Semashev wrote:

...

On Thursday 29 May 2014 22:22:05 Julian Gonggrijp wrote:

...

PROPOSAL

The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:

1. Reduction of dependencies between Boost libraries.

This is a worthy goal, I don't think anyone reasonably disagrees with this.

+1

...

Agree, to a reasonable point. I don't think that solid libraries should be torn apart or unrelated components of different libraries should be mixed together just to reduce dependencies. The cases where this would be beneficial should be discussed with library maintainers.

I think that notion of optional dependencies it also needed to achieve this goal. There are multiple places in Boost where dependencies are intentionally loosened but formally exist.

Right the devil is in the detail.

For example the Math lib is one of those with cyclic dependencies - there's a small core that a few libraries depend on, but which Math also uses to some degree (like lexical_cast).

The question is:

* How many sub libraries does it need to be split into? FP classification is one, constants another, possibly rounding functions one more, then one more for everything else. Grief.

I am not sure the Grief need to be so bad. Are you assuming all modules have to be maintained in separate Git repositories? That does not have to be the case. I will try to explain better, i.e. with other words, what I have tried to say in past discussion on this topic. I will use Boost.Math as an example, but this applies to any Boost Library, maybe in particular to Config which configure dependencies to a large number of technology dependent supporting features. Those features can be seen as modules as well of you follow my line of though. So, let us say Math repository is *not* split in any way but continue more or less as is, but is modularized somehow. What could that look like: First of all, everything that is supporting to the library itself such as tests, examples, docs, etc. are defined as separate modules. I think these already live in their separate sub-directories and have their separate build mechanics. None of this incur dependencies in the Math Library modules themselves. Then, everything in the library itself is split into modules based on some analysis of dependencies and re-engineering deemed needed to reduce those dependencies. The source and headers does not necessarily need to be moved around to make it explicit in the file structure of the repository. That could be at the discretion of library maintainers. However, the library maintainer need to support some means for external tools to identify the content of the modules and their direct dependencies. Such tools could then support various packaging and deployment needs. The granularity of modules may vary a-lot. Not accepting that will create problems. As example, there is some talk of optional headers in this discussion, in my mind that is simply a separate module. It is no need to require such single header modules to be moved into their own library, or to attempt to collect such headers with related dependencies into some common module. That is just turning the problem around, mot really solving anything. To understand where modularizing is going, and thus be able to answer the poll with less ambiguities. I think some fundamentals need to be agreed. Some issues in that regard is at least unclear to me, or I expect there to be other views than mine on the list if we where asked questions, like for example: - What is a module? - What is the relationship between modules and libraries? - What is the relationship between modules and source repositories? - When does a dependency between two modules occur? - What sort of dependencies do we want to prevent occurring? Intentions of those willing to proceed with work on modularizing boost is not clearly stated. References to past discussions are given, but no conclusions about how to proceed. I think that is intentional as some of the process may best be agreed to later. However there is a danger of starting off in the wrong direction. So maybe we should try to agree on some ground rules for modules in boost. Below are my current notions of how the above questions should be answered. I expect other to have other views. ----------- What is a module? ------------- A module can be a single file or a collection of files (e.g.: header and source files) that are hard or impossible to use without the rest of the module. A module is self contained, except that it may have defined dependencies to other modules incurring when it is used by a given tool-set. ---- What is the relationship between modules and libraries? ----- A library has a maintainer individual or group and is maintained in a separate repository from other libraries in Boost. A library may contain many modules. ---- What is the relationship between modules and repositories? ----- Repositories are used to maintain configuration control of a library or some other set of modules in boost. A repository may contain many modules. ---- When does a dependency between two modules occur? ----- The dependency occur when a certain tool processing the dependent module uses or is attempting to use the module it is depending upon. In the latter case we have a failure, which we clearly want to prevent, if that is not feasible, then failures should at least be simple to resolve by identifying the dependency and get it properly deployed onto the system. ---- What sort of dependencies do we want to prevent occurring? ---- Dependencies to modules providing features the user is not requiring. Dependencies to modules using technology the user is not utilizing. -- Bjørn

On 05/31/2014 01:41 PM, Andrey Semashev wrote:

On Saturday 31 May 2014 13:23:56 Bjørn Roald wrote:

...

I am not sure the Grief need to be so bad. Are you assuming all modules have to be maintained in separate Git repositories? That does not have to be the case.

Can you actually download a part of a git repository? If not, I think this kind of defeats the modularization purpose, which is basically to reduce the amount of code you need to download to obtain a minimal part of Boost. E.g. to use Boost.Conversion you would still need to download the whole Boost.Math.

That is assuming you need to use Git as the means to deploy boost Boost.Conversion onto your system. That may be a fair assumption if you are a boost developer, but it need not be the case if you are a user. Besides that I think Conversion already lives in its own Git repo. Further, my way of thinking about modules will not prevent maintainers of Boost.Math or other libraries to split or move modules into multiple repositories. However, forcing a one module in one repository policy will have a lot of issues, and IMHO it will not be feasible to make a clean model of what modules are. -- Bjørn

On 31 May 2014 12:23, Bjørn Roald wrote:

---- What is the relationship between modules and libraries? ----- A library has a maintainer individual or group and is maintained in a separate repository from other libraries in Boost. A library may contain many modules.

We already have multiple libraries in a repository. For example: https://github.com/boostorg/functional/blob/develop/meta/libraries.json These metadata files are only in a few libraries at the moment, but dependency specification could possibly be added to them. Although if it's going to be done for subsets of libraries, it would need to be in a separate file or files.

John Maddock wrote:

On 30/05/2014 10:44, Andrey Semashev wrote:

...

On Thursday 29 May 2014 22:22:05 Julian Gonggrijp wrote:

...

PROPOSAL

The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:

1. Reduction of dependencies between Boost libraries.

This is a worthy goal, I don't think anyone reasonably disagrees with this.

...

Agree, to a reasonable point. I don't think that solid libraries should be torn apart or unrelated components of different libraries should be mixed together just to reduce dependencies. The cases where this would be beneficial should be discussed with library maintainers.

I think that notion of optional dependencies it also needed to achieve this goal. There are multiple places in Boost where dependencies are intentionally loosened but formally exist.

Right the devil is in the detail.

For example the Math lib is one of those with cyclic dependencies - there's a small core that a few libraries depend on, but which Math also uses to some degree (like lexical_cast).

The question is:

* How many sub libraries does it need to be split into? FP classification is one, constants another, possibly rounding functions one more, then one more for everything else. Grief. * How can such a split be done and yet still provide a seemless experience for the user - i.e. I'm strongly in favour of keeping the docs all in one place, and the header directory structure as it is now, while possibly allowing users that are just using say lexical_cast to not have to download the whole Math lib. * Ideally I'd like to keep maintenance overhead as low as possible - the change to the new Git directory structure has been quite disruptive, and everyone involved in maintaining the Math lib has got into "header hell" more than once by editing the wrong file. Splitting the lib makes that even worse.

In general, of course we should not cut dependencies at all costs. From what you say I get the impression Boost.Math might actually be best off as a monolithical submodule.

We also need sensible policies for dealing with optional components - a good example would be libraries that provide serialization support in a separate optional header. The library as such does not require Boost.Serialization, but quite rightly the optional "bridging" support is there. I asked about this last time this topic came up, but I saw no good answer?

I think what you describe here would be a good solution.

Please note: *I still think reducing dependencies is a good idea*, but I'd really like to see some kind of workflow that addresses these issues, otherwise I fear actually making things worse if libraries get harder to maintain at the alter of reduced dependencies.

So far there seems to be quite some support for the general idea of dependency reduction and automated dependency cloning, so I'm willing to write a more detailed proposal that includes (as unintrusive as possible) policies for library maintainers when I find the time. -Julian

This only gives the primary dependencies; it would be nice for the finished product to include the secondary module dependencies as well (without going into specifics what included what, just what module brought up the secondary dependency).

And here's the secondary report on smart_ptr. (My initial version listed modules in alphabetical order, not in discovery order, which was even more stunning as I hadn't the faintest idea why all these modules were deemed a dependency.) Secondary dependencies for smart_ptr: align: adds integer adds static_assert exception: adds smart_ptr adds tuple type_traits: adds mpl adds preprocessor adds static_assert adds typeof utility: adds iterator adds mpl adds preprocessor adds static_assert iterator: adds concept_check adds conversion adds detail adds function_types adds optional mpl: adds detail concept_check: adds parameter conversion: adds array adds container adds math adds numeric/conversion adds range detail: adds predef optional: adds move array: adds functional container: adds functional adds intrusive math: adds algorithm adds atomic adds format adds function adds fusion adds multiprecision adds numeric/ublas adds random adds regex adds test adds tr1 parameter: adds python range: adds algorithm adds functional adds regex algorithm: adds bind function: adds bind fusion: adds *unknown* multiprecision: adds rational numeric/ublas: adds *unknown* adds serialization python: adds bind adds foreach random: adds system test: adds bind adds io adds timer tr1: adds bind adds unordered serialization: adds spirit adds variant timer: adds chrono chrono: adds accumulators adds interprocess adds ratio spirit: adds iostreams adds phoenix adds pool adds proto adds thread accumulators: adds circular_buffer interprocess: adds date_time thread: adds date_time date_time: adds tokenizer

On Fri, May 30, 2014 at 9:02 AM, Peter Dimov wrote:

exception: adds smart_ptr adds tuple

This was the most surprising to me. I assumed, and now really desire, exception to not have any other dependencies.

utility: [snip: Long list]

This is the big offender which results in the whole world being pulled in (exaggeration for effect), and we both use it only for boost::addressof. Glen

...

This only gives the primary dependencies; it would be nice for the finished product to include the secondary module dependencies as well (without going into specifics what included what, just what module brought up the secondary dependency).

And here's the secondary report on smart_ptr. (My initial version listed modules in alphabetical order, not in discovery order, which was even more stunning as I hadn't the faintest idea why all these modules were deemed a dependency.)

A lot of those look like testing dependencies rather than header-usage dependencies? John.

Peter Dimov wrote:

Julian Gonggrijp wrote:

...

PROPOSAL

The following (evolutionary) global changes to Boost should be planned and given priority over any other proposals [e.g. 5], in the following order:

1. Reduction of dependencies between Boost libraries. 2. Simple but effective automation of dependency handling.

We all agree on these goals in principle, but what do you actually propose we need to do? We need to start somewhere. Specifics are needed.

By now it is clear to me that more specifics are needed. I'm looking out for time to work on that.

What I can contribute at the moment is

- a suggestion that we publish a Boost dependency report in HTML format somewhere on boost.org that tells us, and anyone interested, what depends on what and why;

- a program that generates such a report (in plain text) as a starting point (attached).

Thank you for this contribution, Peter! Ideally the HTML report is generated from the latter plaintext, of course. In order for your program to provide input for the automated downloader, I think it would have to provide information at a coarser level of detail, i.e. entire submodules. This is something we can figure out in due time. -Julian

Peter Dimov wrote:

Julian Gonggrijp wrote:

...

Thank you for this contribution, Peter! Ideally the HTML report is generated from the latter plaintext, of course.

The next version of the program I posted has the option of generating HTML directly. You've probably seen the output. If not, do look at a few modules; it's good to have an idea of what our future automated downloader would do when you tell it to download, say, smart_ptr (in the present state of Boost).

Great, I will have a look at it when I'm back in a more programmer-friendly environment. That will take a few more days, please bear with me in the meanwhile. ;-)

...

In order for your program to provide input for the automated downloader, I think it would have to provide information at a coarser level of detail, i.e. entire submodules.

But it does provide the information on a submodule level. It just goes into more detail to explain the dependency, but this additional output can be trivially suppressed. :-)

Excellent!