RFC: Separating Boost.Python from Boost

newer
[next gen future-promise] What to...

Stefan Seefeld

29 May 2015 29 May '15

4:31 p.m.

Hello, as I have mentioned here before, I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost. For avoidance of doubt: my intent is not to remove Boost.Python out of the Boost organization. Rather, I would like to separate the build and release processes. For the sake of keeping the discussion focused, I would like to avoid getting into too many (technical) details (there already is a related discussion on the Boost.Build list). Rather, I would like to know whether people here (and Boost.Python users in particular) have any particular thoughts about this proposal. Any strong reasons not to move forward with this ? Things to consider while doing it ? Etc. (My hope is that if the operation succeeds, other Boost libraries may want to do the same, which would help Boost in the long term to become more manageable.) Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...

Show replies by date

Robert Ramey

29 May 29 May

6:04 p.m.

On 5/29/15 9:31 AM, Stefan Seefeld wrote:

...

Hello,

as I have mentioned here before, I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost.

It's not clear to me what this means. Does this mean altering Boost.Python so that it has no dependencies on other parts of boost? That is that the headers in Boost.Python include no headers from

...

For avoidance of doubt: my intent is not to remove Boost.Python out of the Boost organization. Rather, I would like to separate the build and release processes.

Again, it's not clear what "separate the build and release processes" means. Does this mean the ability to "release (distribute ?) Boost.Python without waiting for a boost release? If this is what you mean, the question really has nothing to do with Boost.Python but rather any Boost library. It's an interesting discussion Robert Ramey

Stefan Seefeld

6:12 p.m.

On 29/05/15 02:04 PM, Robert Ramey wrote:

...

On 5/29/15 9:31 AM, Stefan Seefeld wrote:

...
Hello,

as I have mentioned here before, I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost.

It's not clear to me what this means. Does this mean altering Boost.Python so that it has no dependencies on other parts of boost? That is that the headers in Boost.Python include no headers from

As I mentioned, all I want is being able to build and release Boost.Python independently. No changes to the (C++) code are involved to achieve this. (That doesn't mean that the dependencies on the source code level shouldn't be cleaned up; but that's an orthogonal effort.)

...

How would you plan to do this? A fork of Boost.Python, or unilaterally make changes in the current version of Boost.Python?

I'm already experimenting on a fork (https://github.com/stefanseefeld/boost.python/tree/standalone). I'm proposing to push those changes (once stable) to the official Boost.Python. (Not sure what you mean by "unilaterally". The very reason I'm bringing the subject up for discussion is because I want to build consensus. :-) )

...

If that's what you mean, wouldn't mean copying a bunch of code from other parts of boost into boost python? That would have lot's of interesting implications.

Again, no code changes are involved. Right now Boost.Python is built as part of a Boost build, and likewise is released as part of a Boost release. All I'm suggesting is to build and release Boost.Python stand-alone, against an external Boost installation.

...

...
For avoidance of doubt: my intent is not to remove Boost.Python out of the Boost organization. Rather, I would like to separate the build and release processes.

Again, it's not clear what "separate the build and release processes" means. Does this mean the ability to "release (distribute ?) Boost.Python without waiting for a boost release?

Yes.

...

If this is what you mean, the question really has nothing to do with Boost.Python but rather any Boost library. It's an interesting discussion

Robert Ramey

Stefan -- ...ich hab' noch einen Koffer in Berlin...

Nevin Liber

6:26 p.m.

On 29 May 2015 at 13:12, Stefan Seefeld wrote:

...

Again, no code changes are involved. Right now Boost.Python is built as part of a Boost build, and likewise is released as part of a Boost release. All I'm suggesting is to build and release Boost.Python stand-alone, against an external Boost installation.

Does that mean Boost.Python would no longer be part of a Boost release? -- Nevin ":-)" Liber mailto:nevin@eviloverlord.com (847) 691-1404

Stefan Seefeld

6:32 p.m.

On 29/05/15 02:26 PM, Nevin Liber wrote:

...

On 29 May 2015 at 13:12, Stefan Seefeld wrote:

...
Again, no code changes are involved. Right now Boost.Python is built as part of a Boost build, and likewise is released as part of a Boost release. All I'm suggesting is to build and release Boost.Python stand-alone, against an external Boost installation.

Does that mean Boost.Python would no longer be part of a Boost release?

That would be the goal, yes. (Boost.Python doesn't have any dependencies on recent Boost additions, so I expect it to be very stable with respect to ongoing development in the rest of the Boost code.) Stefan -- ...ich hab' noch einen Koffer in Berlin...

Pete Bartlett

6:53 p.m.

...

On 29 May 2015, at 19:32, Stefan Seefeld wrote:

...
On 29/05/15 02:26 PM, Nevin Liber wrote:

...
On 29 May 2015 at 13:12, Stefan Seefeld wrote:

Again, no code changes are involved. Right now Boost.Python is built as part of a Boost build, and likewise is released as part of a Boost release. All I'm suggesting is to build and release Boost.Python stand-alone, against an external Boost installation. Does that mean Boost.Python would no longer be part of a Boost release?

That would be the goal, yes. (Boost.Python doesn't have any dependencies on recent Boost additions, so I expect it to be very stable with respect to ongoing development in the rest of the Boost code.)

Stefan

As a corporate user of Boost and Boost.python on the face of it this a bad change for me. Right now Boost is one entity so from a legal and various bureaucratic points of view I can bring boost through the door in one fell swoop (I then apply another quality filter before it a particular library is used in anger). Multiple versions has the potential for headaches. Now I know none of that is your problem but you did ask for comments and I am sure others will have the same issue. I realise a fully modular Boost had its advantages but I am hopeful it will sit alongside a big fat integration-tested all of Boost distro too... Pete

Nevin Liber

7:35 p.m.

On 29 May 2015 at 13:32, Stefan Seefeld wrote:

...

On 29/05/15 02:26 PM, Nevin Liber wrote:

...
On 29 May 2015 at 13:12, Stefan Seefeld wrote:

...
Again, no code changes are involved. Right now Boost.Python is built as part of a Boost build, and likewise is released as part of a Boost release. All I'm suggesting is to build and release Boost.Python stand-alone, against an external Boost installation.

Does that mean Boost.Python would no longer be part of a Boost release?

That would be the goal, yes. (Boost.Python doesn't have any dependencies on recent Boost additions, so I expect it to be very stable with respect to ongoing development in the rest of the Boost code.)

I am strongly against breaking up the Boost release. I also think such a fundamental change has to go through the Steering Committee. -- Nevin ":-)" Liber mailto:nevin@eviloverlord.com (847) 691-1404

Stefan Seefeld

9:25 p.m.

On 29/05/15 03:35 PM, Nevin Liber wrote:

...

I am strongly against breaking up the Boost release.

Just for context: We recently had a discussion (with Niall) where the idea of a "Boost distribution" came up. In such a scenario, Boost libraries could still follow their individual release schedule, but would be made available through a single distribution channel (which could also assume some validation tasks to test inter-version compatibility. I think this would be an interesting model to consider, as it adds the best of both worlds: independence an autonomy of projects, and convenience for end users.

...

I also think such a fundamental change has to go through the Steering Committee.

Well, I beg to differ. It's the projects' maintainers / developers who have to do the actual work, so ultimately it's their decision where to put their effort. Stefan -- ...ich hab' noch einen Koffer in Berlin...

Rob Stewart

10:03 p.m.

On May 29, 2015 5:25:14 PM EDT, Stefan Seefeld wrote:

...

On 29/05/15 03:35 PM, Nevin Liber wrote:

...
I am strongly against breaking up the Boost release.

Just for context: We recently had a discussion (with Niall) where the idea of a "Boost distribution" came up. In such a scenario, Boost libraries could still follow their individual release schedule, but would be made available through a single distribution channel (which could also assume some validation tasks to test inter-version compatibility. I think this would be an interesting model to consider, as it adds the best of both worlds: independence an autonomy of projects, and convenience for end users.

That single distribution channel is the monolithic Boost release. There are no plans to replace that anytime soon.

...

...
I also think such a fundamental change has to go through the Steering Committee.

Well, I beg to differ. It's the projects' maintainers / developers who have to do the actual work, so ultimately it's their decision where to put their effort.

If your changes have the effect of removing Boost.Python from Boost releases, then they have broader implications than the convenience or wishes of the maintainers. ___ Rob (Sent from my portable computation engine)

Stefan Seefeld

10:08 p.m.

On 29/05/15 06:03 PM, Rob Stewart wrote:

...

On May 29, 2015 5:25:14 PM EDT, Stefan Seefeld wrote:

...
...
I am strongly against breaking up the Boost release. Just for context: We recently had a discussion (with Niall) where the idea of a "Boost distribution" came up. In such a scenario, Boost

On 29/05/15 03:35 PM, Nevin Liber wrote: libraries could still follow their individual release schedule, but would be made available through a single distribution channel (which could also assume some validation tasks to test inter-version compatibility. I think this would be an interesting model to consider, as it adds the best of both worlds: independence an autonomy of projects, and convenience for end users. That single distribution channel is the monolithic Boost release. There are no plans to replace that anytime soon.

For whom are you speaking here when you say "there are no plans" ?

...

...
...
I also think such a fundamental change has to go through the Steering Committee. Well, I beg to differ. It's the projects' maintainers / developers who have to do the actual work, so ultimately it's their decision where to put their effort. If your changes have the effect of removing Boost.Python from Boost releases, then they have broader implications than the convenience or wishes of the maintainers.

Sure, I didn't say they don't have implications for anyone. (I wouldn't have started the discussion if they hadn't.) However, only the people doing the actual work can chose where they put their effort. So you shouldn't discount them when talking about decisions to be made (or not). Stefan -- ...ich hab' noch einen Koffer in Berlin...

Peter Dimov

10:31 p.m.

Stefan Seefeld wrote:

...

Sure, I didn't say they don't have implications for anyone. (I wouldn't have started the discussion if they hadn't.) However, only the people doing the actual work can chose where they put their effort. So you shouldn't discount them when talking about decisions to be made (or not).

As one data point, ASIO is an independent library that has its own releases but is still part of the Boost release.

Stefan Seefeld

10:40 p.m.

On 29/05/15 06:31 PM, Peter Dimov wrote:

...

Stefan Seefeld wrote:

...
Sure, I didn't say they don't have implications for anyone. (I wouldn't have started the discussion if they hadn't.) However, only the people doing the actual work can chose where they put their effort. So you shouldn't discount them when talking about decisions to be made (or not).

As one data point, ASIO is an independent library that has its own releases but is still part of the Boost release.

What do you mean by "independent" ? How does an ASIO release relate to a Boost release ? Stefan -- ...ich hab' noch einen Koffer in Berlin...

Peter Dimov

11:22 p.m.

Stefan Seefeld wrote: On 29/05/15 06:31 PM, Peter Dimov wrote:

...

...
As one data point, ASIO is an independent library that has its own releases but is still part of the Boost release.

What do you mean by "independent" ? How does an ASIO release relate to a Boost release ?

https://think-async.com/Asio/AsioAndBoostAsio It's an interesting relationship, but ASIO has existed before becoming part of Boost, which is probably why Chris still maintains both forks. Boost.Python doesn't need any source changes.

Niall Douglas

30 May 30 May

12:48 p.m.

On 30 May 2015 at 2:22, Peter Dimov wrote:

...

Stefan Seefeld wrote: On 29/05/15 06:31 PM, Peter Dimov wrote:

...
...
As one data point, ASIO is an independent library that has its own releases but is still part of the Boost release.

What do you mean by "independent" ? How does an ASIO release relate to a Boost release ?

https://think-async.com/Asio/AsioAndBoostAsio

Standalone ASIO is also the proposed reference Networking TS implementation. I am unsure if that part will be contributed to Boost.

...

It's an interesting relationship, but ASIO has existed before becoming part of Boost, which is probably why Chris still maintains both forks.

Some corporations have explicit blanket bans for all Boost libraries except X, Y and Z due to the header only problem. Standalone ASIO explicitly does not mention Boost in its name to work around that fact. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Andrey Semashev

29 May 29 May

10:32 p.m.

On Friday 29 May 2015 18:08:26 Stefan Seefeld wrote:

...

On 29/05/15 06:03 PM, Rob Stewart wrote:

...
If your changes have the effect of removing Boost.Python from Boost releases, then they have broader implications than the convenience or wishes of the maintainers.

Sure, I didn't say they don't have implications for anyone. (I wouldn't have started the discussion if they hadn't.) However, only the people doing the actual work can chose where they put their effort. So you shouldn't discount them when talking about decisions to be made (or not).

Although I agree with you that it is the library maintainers who choose where to put their effort, I would still like to ask you to provide means of distributing your library as part of monolithic Boost (as long as it is the current distribution model). That includes supporting distributing and building/testing your library as a part of the Boost tree, as well as probably communicating with the release managers in case if their release/docs building scripts need updating. You have to understand that not everyone is going to switch to the modularized distribution model right away, especially since it is only Boost.Python that gets modularized for now. Ideally, I would suggest making the "standalone" build scripts more or less separate, without affecting the current integration into Boost infrastructure. If your plan is to remove support for building your library as a part of monolithic Boost then, at this point, this would basically mean removing the library from Boost distribution. I'm sure this would be a painful change for users, so forking the library may be more preferable in this case.

Stefan Seefeld

10:38 p.m.

On 29/05/15 06:32 PM, Andrey Semashev wrote:

...

...
...
If your changes have the effect of removing Boost.Python from Boost releases, then they have broader implications than the convenience or wishes of the maintainers. Sure, I didn't say they don't have implications for anyone. (I wouldn't have started the discussion if they hadn't.) However, only the people doing the actual work can chose where they put

On 29/05/15 06:03 PM, Rob Stewart wrote: their effort. So you shouldn't discount them when talking about decisions to be made (or not). Although I agree with you that it is the library maintainers who choose where to put their effort, I would still like to ask you to provide means of distributing your library as part of monolithic Boost (as long as it is the current distribution model). That includes supporting distributing and building/testing your library as a part of the Boost tree, as well as probably communicating with the release managers in case if their release/docs building

On Friday 29 May 2015 18:08:26 Stefan Seefeld wrote: scripts need updating. You have to understand that not everyone is going to switch to the modularized distribution model right away, especially since it is only Boost.Python that gets modularized for now. Ideally, I would suggest making the "standalone" build scripts more or less separate, without affecting the current integration into Boost infrastructure.

That's a fair concern, and my initial plans were indeed to support both, and in-tree as well as a stand-alone build. I'll see how hard this is to achieve.

...

If your plan is to remove support for building your library as a part of monolithic Boost then, at this point, this would basically mean removing the library from Boost distribution. I'm sure this would be a painful change for users, so forking the library may be more preferable in this case.

Right, understood. The problem I expect is that, as long as both modes are enabled, nothing will ever change, so a little bit of coercion is a Good Thing. ;-) The ultimate goal I have is for Boost to become an umbrella organization of individual projects (typically one per library), which all prepare their releases independently and autonomously. Some common infrastructure would allow to validate their inter-operability, and thus make it easier to establish a "boost distribution", where users can choose the components from they really need. Of course that's not a decision I (or anyone else for that matter) can take on alone. It's something to enable in each individual library, though, and eventually to agree upon as a community. Stefan -- ...ich hab' noch einen Koffer in Berlin...

Rob Stewart

11:41 p.m.

On May 29, 2015 6:38:59 PM EDT, Stefan Seefeld wrote:

...

...
On Friday 29 May 2015 18:08:26 Stefan Seefeld wrote:

...
On 29/05/15 06:03 PM, Rob Stewart wrote:

...
If your changes have the effect of removing Boost.Python from Boost releases, then they have broader implications than the convenience or wishes of the maintainers. Sure, I didn't say they don't have implications for anyone. (I wouldn't have started the discussion if they hadn't.) However, only the people doing the actual work can chose where they

On 29/05/15 06:32 PM, Andrey Semashev wrote: put

...
...
their effort. So you shouldn't discount them when talking about decisions to be made (or not).

If your decision affects Boost, then I should indeed speak up.

...

...
If your plan is to remove support for building your library as a part of monolithic Boost then, at this point, this would basically mean removing the library from Boost distribution. I'm sure this would be a painful change for users, so forking the library may be more preferable in this case.

Right, understood. The problem I expect is that, as long as both modes are enabled, nothing will ever change, so a little bit of coercion is a Good Thing. ;-)

...

From the tone and direction of your posts, you have decided that Boost will move away from monolithic releases and that sooner is better than later. There are many interested in getting away from monolithic releases, but not all are convinced. The onus, then, is on those that want the change to create the infrastructure to produce user-directed distributions. Once those are available, user choices should indicate if such distributions are useful and wanted. If that's the case, Boost can plan a transition from monolithic releases.

...

The ultimate goal I have is for Boost to become an umbrella organization of individual projects (typically one per library), which all prepare their releases independently and autonomously. Some common infrastructure would allow to validate their inter-operability, and thus make it easier to establish a "boost distribution", where users can choose the components from they really need.

Of course that's not a decision I (or anyone else for that matter) can take on alone. It's something to enable in each individual library, though, and eventually to agree upon as a community.

I take your plan to be to remake Boost according to your wishes, dragging the community along for the ride. That disturbs me. If I've read too much into your posts, please clarify what I've misunderstood. ___ Rob (Sent from my portable computation engine)

Stefan Seefeld

11:54 p.m.

On 29/05/15 07:41 PM, Rob Stewart wrote:

...

I take your plan to be to remake Boost according to your wishes, dragging the community along for the ride. That disturbs me. If I've read too much into your posts, please clarify what I've misunderstood.

Sorry for being so unclear. I have no plans to "remake Boost". I have a plan to change the way Boost.Python is built and released (though even that is more a proposal at this point than anything else). The reason I have started this discussion (a "Request for Comment", in fact), is to get a sense of how such a change would be received both by other developers (Boost.Python as well as other Boost libraries), as well as Boost users, and have a constructive discussion about what our (collective) goals are and how to get there. This is not a decision anyone can take alone. But those who do the actual work should certainly have more weight than others. That's just the way Open Source works. Stefan -- ...ich hab' noch einen Koffer in Berlin...

Oleg Grunin

11:52 p.m.

On 5/29/15 2:32 PM, Stefan Seefeld wrote:

...

On 29/05/15 02:26 PM, Nevin Liber wrote:

...
On 29 May 2015 at 13:12, Stefan Seefeld wrote:

...
Again, no code changes are involved. Right now Boost.Python is built as part of a Boost build, and likewise is released as part of a Boost release. All I'm suggesting is to build and release Boost.Python stand-alone, against an external Boost installation.

Does that mean Boost.Python would no longer be part of a Boost release?

That would be the goal, yes. (Boost.Python doesn't have any dependencies on recent Boost additions, so I expect it to be very stable with respect to ongoing development in the rest of the Boost code.)

Stefan

In what sense then, will Boost.Python, decoupled from Boost, remain Boost.Python and not SmthElse.Python?

Stefan Seefeld

30 May 30 May

1:19 a.m.

On 29/05/15 07:52 PM, Oleg Grunin wrote:

...

On 5/29/15 2:32 PM, Stefan Seefeld wrote:

...
On 29/05/15 02:26 PM, Nevin Liber wrote:

...
On 29 May 2015 at 13:12, Stefan Seefeld wrote:

...
Again, no code changes are involved. Right now Boost.Python is built as part of a Boost build, and likewise is released as part of a Boost release. All I'm suggesting is to build and release Boost.Python stand-alone, against an external Boost installation.

Does that mean Boost.Python would no longer be part of a Boost release?

That would be the goal, yes. (Boost.Python doesn't have any dependencies on recent Boost additions, so I expect it to be very stable with respect to ongoing development in the rest of the Boost code.)

Stefan

In what sense then, will Boost.Python, decoupled from Boost, remain Boost.Python and not SmthElse.Python?

That does entirely depend on what we mean by "Boost". The way I would like to see things evolve, Boost would become an umbrella organization hosting certain projects (compare this to the Apache foundation at http://www.apache.org/, for example). Boost would not be a package that gets released once or twice per year. So, in that sense, I expect that Boost.Python would be a "Boost member project" which is released independently from other projects also hosted by the Boost organization. Does that make sense ? Stefan -- ...ich hab' noch einen Koffer in Berlin...

Sergey Cheban

31 May 31 May

9:25 p.m.

30.05.2015 4:19, Stefan Seefeld пишет:

>> In what sense then, will Boost.Python, decoupled from Boost, remain
>> Boost.Python and not SmthElse.Python?
> That does entirely depend on what we mean by "Boost".
For me, the Boost is a set of libraries that:
1. Have the same Boost license.
2. Are accepted by the Boost community.
3. Use the same naming convention (currently, it is the boost:: 
namespace and the boost/ prefix for include files).
4. Are tested and work well together (i mean well-known versions of the 
libraries).
5. Can be built and installed using the same [bjam] engine.
6. Are distributed from boost.org.

A good example of such a library is boost::spirit. Note that:
1. It is developed separately (has its own newsgroups and site).
2. It has its own version numbering. Actually, boost contains two 
versions of this library, 1.8.x and 2.5.x, side by side.



-- 
Best regards,
Sergey Cheban

Robert Ramey

29 May 29 May

7:24 p.m.

On 5/29/15 11:12 AM, Stefan Seefeld wrote:

...

On 29/05/15 02:04 PM, Robert Ramey wrote:

...
On 5/29/15 9:31 AM, Stefan Seefeld wrote:

...

As I mentioned, all I want is being able to build and release Boost.Python independently. No changes to the (C++) code are involved to achieve this.

...

...
How would you plan to do this? A fork of Boost.Python, or unilaterally make changes in the current version of Boost.Python?

I'm already experimenting on a fork (https://github.com/stefanseefeld/boost.python/tree/standalone). I'm proposing to push those changes (once stable) to the official Boost.Python. (Not sure what you mean by "unilaterally". The very reason I'm bringing

OK - I've looked at the fork and see what you want to do and I still have a few questions; a) I perused the python library source code and it includes only couple of boost header files. So it's not really dependent on the rest of boost at all. Not a good or bad thing - just an observation. b) I see that your proposal just involves changes in the jamfiles eliminating dependencies on some other jamfiles. Actually I didn't even realize that a libraries jamfiles actually depended upon the root. Now that I realize this I see the logic of it. If you can do this I don't see any downside to it. It doesn't conflict with anything else.

...

All I'm suggesting is to build and release Boost.Python stand-alone, against an external Boost installation.

c) It seems to me that you wouldn't even need boost installed to do this. Again not a good or bad thing.

...

...
...
For avoidance of doubt: my intent is not to remove Boost.Python out of the Boost organization.

d) regardless of your intent, that decision to include or exclude boost python in a "release" would be a separate question. I don't think you could prevent any boost library from being included in the "boost release"

...

...
Does this mean the ability to "release (distribute ?) Boost.Python without waiting for a boost release?

Yes.

Actually, this what we want to be able to do for all boost libraries. See the related initiative proposed in the steering committee mailing list. This has been proposed as a long term goal. But it seems (as usual) that current events out flank long term plans. While you're at it, Why not include CMake files so that the build/test of Boost python be totally decoupled from from the boost build system. I did this for the boost serialization library so in your world this would look like the ability to deliver the latest master version of the boost serialization library independently of the next release. In fact, that is exactly what I recommend to users who have detected errors in a couple of corner cases of the library which have since been fixed. Basically, you should think bigger. Robert Ramey

Peter Dimov

7:57 p.m.

Robert Ramey wrote:

...

a) I perused the python library source code and it includes only couple of boost header files. So it's not really dependent on the rest of boost at all. Not a good or bad thing - just an observation.

That's what boostdep is for. "boostdep python" will tell you what it includes. http://www.pdimov.com/tmp/report-develop-08112c5/python.html

Robert Ramey

8:25 p.m.

On 5/29/15 12:57 PM, Peter Dimov wrote:

...

Robert Ramey wrote:

...
a) I perused the python library source code and it includes only couple of boost header files. So it's not really dependent on the rest of boost at all. Not a good or bad thing - just an observation.

That's what boostdep is for. "boostdep python" will tell you what it includes.

http://www.pdimov.com/tmp/report-develop-08112c5/python.html

very good - I didn't think of that. I guess it pretty well proves taht I was wrong in thinking that python could be decoupled from other boost source code. So it seems that this "independence" really just means making the bjam files not dependent on the boost root ones. I'm not sure that I see the point to or the value in this. But it still seems to me that it doesn't change anything. Robert Ramey

Stefan Seefeld

9:21 p.m.

On 29/05/15 03:24 PM, Robert Ramey wrote:

...

On 5/29/15 11:12 AM, Stefan Seefeld wrote:

...
...
Does this mean the ability to "release (distribute ?) Boost.Python without waiting for a boost release?

Yes.

Actually, this what we want to be able to do for all boost libraries. See the related initiative proposed in the steering committee mailing list. This has been proposed as a long term goal. But it seems (as usual) that current events out flank long term plans.

Yeah, and this is why I'm taking the initiative, to get unstuck again.

...

While you're at it,

No, no, no ! :-) Seriously, there is clearly a lot that could be done "while we are at it", but it's exactly this kind of thought process that ultimately causes things to stall. So I'd rather not add additional concerns, especially if related to decisions about tools. Let's *please* keep these things separate. Stefan -- ...ich hab' noch einen Koffer in Berlin...

Edward Diener

7:53 p.m.

On 5/29/2015 12:31 PM, Stefan Seefeld wrote:

...

Hello,

as I have mentioned here before, I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost. For avoidance of doubt: my intent is not to remove Boost.Python out of the Boost organization. Rather, I would like to separate the build and release processes. For the sake of keeping the discussion focused, I would like to avoid getting into too many (technical) details (there already is a related discussion on the Boost.Build list). Rather, I would like to know whether people here (and Boost.Python users in particular) have any particular thoughts about this proposal. Any strong reasons not to move forward with this ? Things to consider while doing it ? Etc.

(My hope is that if the operation succeeds, other Boost libraries may want to do the same, which would help Boost in the long term to become more manageable.)

Does Boost.python have dependencies on any other Boost libraries ? If so how do you ensure that the standalone version you propose will work with whatever dependencies it has of various Boost distributions which exist ? Does any other Boost library depend on Boost.python ? If so how do you ensure that another Boost library will work with your proposed standalone version of Boost.python ?

Niall Douglas

9:35 p.m.

On 29 May 2015 at 12:31, Stefan Seefeld wrote:

...

Rather, I would like to know whether people here (and Boost.Python users in particular) have any particular thoughts about this proposal. Any strong reasons not to move forward with this ? Things to consider while doing it ? Etc.

This is exactly what proposed Boost.APIBind is designed for: allow a legacy Boost library to both exist as part of monolithic Boost, and as a standalone library with no Boost dependencies (with the C++ 11 STL swapped in instead). Identical code base can do both targets. Have a look at https://svn.boost.org/trac/boost/wiki/BestPracticeHandbook#a16.COUPLIN G:Considerallowingyourlibraryuserstodependencyinjectyourdependencieson otherlibraries. If that whets your appetite, have a look at my C++ Now Presentation on APIBind: https://github.com/boostcon/cppnow_presentations_2015/raw/master/files /A-review-of-Cxx-11-14-only-Boost-libraries-Fiber-AFIO-DI-and-APIBind. pdf The video of that talk is probably also very useful to watch, but I don't think they are posted yet. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Robert Ramey

30 May 30 May

3:11 a.m.

On 5/29/15 9:31 AM, Stefan Seefeld wrote:

...

Hello,

as I have mentioned here before, I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost.

I think this explanation has totally confused things. I get from the thread and by looking at your fork that you intend to make a fork of the bjam files which will depend less on the jamfile in boost root. As far as I can tell, that's all your going to do. So you could build boost python with having the upper level bjam files. That is all I see here. I don't see anything that would inhibit the modified version from being distributed with the rest of boost as it currently does. I don't see much point to this as it stands as it would still require all the other boost headers that this library depends upon - 5 levels. Why would anyone want to use this? and for what? Now if you wanted to make a system which would download and package any subset of boost libraries that might be interesting. But I'm just not seeing what one hopes to accomplish here. As an aside - A lot of confusion is generated by the fact that boost does multiple things here. It reviews/certifies libraries, it tests librares and it distributes a complete set of libraries. There is no reason that all these functions need to be done by the same entity - in fact I don't think that's the most expedient way to do it. I want to see us move in the direction of decoupling these three functions. Robert Ramey

Bjorn Reese

9:43 a.m.

On 05/30/2015 05:11 AM, Robert Ramey wrote:

...

So you could build boost python with having the upper level bjam files. [...] Why would anyone want to use this? and for what?

I do not know why anyone want to do that for Boost.Python, but this problem is of a general nature. Every author of a new Boost library has to go through the "birth-pain" of getting their library to build with Boost.Build while it is not part of monolithic Boost yet. Currently, the easiest way is to checkout the Boost codebase and then copy your library in there. A more involved alternative is to cheat Boost.Build into believing that your library is part of the Boost code base by adding lots of symbolic links all around. Neither solution is feasible if you want other projects to use your library. Being able to build such proposed libraries without upper level bjam files would help a lot. This could lower the pratical barriers of entry for new Boost libraries.

Vladimir Prus

12:07 p.m.

On 5/30/2015 12:43 PM, Bjorn Reese wrote:

...

On 05/30/2015 05:11 AM, Robert Ramey wrote:

...
So you could build boost python with having the upper level bjam files. [...] Why would anyone want to use this? and for what?

I do not know why anyone want to do that for Boost.Python, but this problem is of a general nature.

Every author of a new Boost library has to go through the "birth-pain" of getting their library to build with Boost.Build while it is not part of monolithic Boost yet.

Currently, the easiest way is to checkout the Boost codebase and then copy your library in there. A more involved alternative is to cheat Boost.Build into believing that your library is part of the Boost code base by adding lots of symbolic links all around. Neither solution is feasible if you want other projects to use your library.

Being able to build such proposed libraries without upper level bjam files would help a lot. This could lower the pratical barriers of entry for new Boost libraries.

If that's perceived a significant problem, I can try to improve that - like, I could try making upcoming Boost.DLL testable without copy-into-boost? Thanks, Volodya

Andrey Semashev

6:06 p.m.

On Saturday 30 May 2015 15:07:15 Vladimir Prus wrote:

...

On 5/30/2015 12:43 PM, Bjorn Reese wrote:

...
On 05/30/2015 05:11 AM, Robert Ramey wrote:

...
So you could build boost python with having the upper level bjam files.

[...]

...
Why would anyone want to use this? and for what?

I do not know why anyone want to do that for Boost.Python, but this problem is of a general nature.

Every author of a new Boost library has to go through the "birth-pain" of getting their library to build with Boost.Build while it is not part of monolithic Boost yet.

Currently, the easiest way is to checkout the Boost codebase and then copy your library in there. A more involved alternative is to cheat Boost.Build into believing that your library is part of the Boost code base by adding lots of symbolic links all around. Neither solution is feasible if you want other projects to use your library.

Being able to build such proposed libraries without upper level bjam files would help a lot. This could lower the pratical barriers of entry for new Boost libraries.

If that's perceived a significant problem, I can try to improve that - like, I could try making upcoming Boost.DLL testable without copy-into-boost?

When I was developing Boost.Log before it got accepted and merged into Boost, I was asked on multiple occasions about how one could build and use the library outside of a Boost distribution. Since I'm not very good at Boost.Build my recommendation had always been "copy into Boost". If there was a simple way to just build the library as a standalone project that would certainly help.

Niall Douglas

1:25 p.m.

On 30 May 2015 at 11:43, Bjorn Reese wrote:

...

I do not know why anyone want to do that for Boost.Python, but this problem is of a general nature.

Every author of a new Boost library has to go through the "birth-pain" of getting their library to build with Boost.Build while it is not part of monolithic Boost yet.

Currently, the easiest way is to checkout the Boost codebase and then copy your library in there. A more involved alternative is to cheat Boost.Build into believing that your library is part of the Boost code base by adding lots of symbolic links all around. Neither solution is feasible if you want other projects to use your library.

I'm a bit surprised you say this. AFIO requires exactly one symlink into libs/. Modular Boost takes care of all the rest if your Boost library has the right directory structure. On Windows, enabling normal users to create symlinks also gets rid of any weird build failure problems. Or you simply copy afio into libs/afio and be done with it. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Bjorn Reese

1:47 p.m.

On 05/30/2015 03:25 PM, Niall Douglas wrote:

...

I'm a bit surprised you say this. AFIO requires exactly one symlink into libs/. Modular Boost takes care of all the rest if your Boost library has the right directory structure.

If you want to run your own tests then you cd into your test directory and call b2 from there -- likewise if you want to build your own doc. How can you get that to work with a single symbolic link?

Peter Dimov

1:54 p.m.

Niall Douglas wrote:

...

On 30 May 2015 at 11:43, Bjorn Reese wrote:

...

...
I do not know why anyone want to do that for Boost.Python, but this problem is of a general nature.

Every author of a new Boost library has to go through the "birth-pain" of getting their library to build with Boost.Build while it is not part of monolithic Boost yet.

Currently, the easiest way is to checkout the Boost codebase and then copy your library in there. A more involved alternative is to cheat Boost.Build into believing that your library is part of the Boost code base by adding lots of symbolic links all around. Neither solution is feasible if you want other projects to use your library.

I'm a bit surprised you say this. AFIO requires exactly one symlink into libs/. Modular Boost takes care of all the rest if your Boost library has the right directory structure.

Suppose you have Boost in C:\boost-1.58.0, and your proposed Boost.Pumpkin is in C:\projects\pumpkin. If you symlink C:\boost-1.58.0\libs\pumpkin to C:\projects\pumpkin, you'll be able to do this: C:\boost-1.58.0\libs\pumpkin\test>b2 and it will run the tests. But what I think Bjorn wants is this: C:\projects\pumpkin\test>b2 This presumably would involve a Jamroot either at C:\projects\pumpkin or at C:\projects that would point to C:\boost-1.58.0.

...

On Windows, enabling normal users to create symlinks also gets rid of any weird build failure problems. Or you simply copy afio into libs/afio and be done with it.

For directories, you can use junctions on Windows, mklink/j.

Bjorn Reese

2:18 p.m.

On 05/30/2015 03:54 PM, Peter Dimov wrote:

...

If you symlink C:\boost-1.58.0\libs\pumpkin to C:\projects\pumpkin, you'll be able to do this:

C:\boost-1.58.0\libs\pumpkin\test>b2

and it will run the tests. But what I think Bjorn wants is this:

C:\projects\pumpkin\test>b2

I would be content if either worked for me, but in both cases I get the following error on Linux: error: Did not find Jamfile.jam or Jamroot.jam in any parent directory. I also get this for a symlinked AFIO.

Niall Douglas

2:32 p.m.

On 30 May 2015 at 16:18, Bjorn Reese wrote:

...

...
If you symlink C:\boost-1.58.0\libs\pumpkin to C:\projects\pumpkin, you'll be able to do this:

C:\boost-1.58.0\libs\pumpkin\test>b2

and it will run the tests. But what I think Bjorn wants is this:

C:\projects\pumpkin\test>b2

I would be content if either worked for me, but in both cases I get the following error on Linux:

error: Did not find Jamfile.jam or Jamroot.jam in any parent directory.

I also get this for a symlinked AFIO.

I understand the problem now. Before modular boost you could run b2 from within a symlinked afio, and it worked. Since modular boost, b2 only works right with symlinked libraries if you call it from the top of the boost tree e.g. ./b2 libs/afio/test --link-test -j 8 I also have the peculiarity that to generate docs I must do this: cd boost/doc ../b2 ../libs/afio/doc ... otherwise output appears in the wrong directories, and I have no idea why. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Peter Dimov

2:59 p.m.

Niall Douglas wrote:

...

Since modular boost, b2 only works right with symlinked libraries if you call it from the top of the boost tree e.g.

./b2 libs/afio/test --link-test -j 8

Sounds like b2 does realpath on the current directory somewhere.

Robert Ramey

5:41 p.m.

On 5/30/15 2:43 AM, Bjorn Reese wrote:

...

On 05/30/2015 05:11 AM, Robert Ramey wrote:

...
So you could build boost python with having the upper level bjam files. [...] Why would anyone want to use this? and for what?

I do not know why anyone want to do that for Boost.Python, but this problem is of a general nature.

Every author of a new Boost library has to go through the "birth-pain" of getting their library to build with Boost.Build while it is not part of monolithic Boost yet.

Currently, the easiest way is to checkout the Boost codebase and then copy your library in there.

I had thought the the procedure for testing a new/proposed library would be: a) clone the whole of boost on one's local machine b) build b2 according to the "bootstrap" instructions c) paste in my new/proposed library d) run b2 headers d) build all the libraries according to the "Getting Started" instructions. This might not build my pasted in library which is fine e) cd to mylibrary/test and run b2 toolset=... etc (or in my case run library status...) Then one should have one's library built and tested. Hmm I admit I haven't actually done exactly this. It just never occurred to me that it wouldn't work. Can I assume that works or not? In addition, I would expect that if I go to boost root and run b2 it would build my new library as well. That is I would hope/assume that b2 would build all the libraries in the lib subdirectory rather than depending on some magic list inside of the the jamfile at the root. I know I could investigate it on my own, but it's easier just to ask here while making my point that this is the way it "should" work. So in my world, the procededure is past in my new library, re-run b2 headers, and run the library tests - which build the library if necessary. That seems very reasonable, understandable and easily doable if it's not setup like that already. We absolutely have to have such a facility to support the review process. Finally, I would like to support the idea that a prospective library could just support CMake. In this case the lack of a Build directory in the library would mean that the global b2 ignores it. The CMake could be run from the local library directory. There would be some other issues but we'll leave that for another time. Robert Ramey

Edward Diener

6:15 p.m.

On 5/30/2015 1:41 PM, Robert Ramey wrote:

...

On 5/30/15 2:43 AM, Bjorn Reese wrote:

...
On 05/30/2015 05:11 AM, Robert Ramey wrote:

...
So you could build boost python with having the upper level bjam files. [...] Why would anyone want to use this? and for what?

I do not know why anyone want to do that for Boost.Python, but this problem is of a general nature.

Every author of a new Boost library has to go through the "birth-pain" of getting their library to build with Boost.Build while it is not part of monolithic Boost yet.

Currently, the easiest way is to checkout the Boost codebase and then copy your library in there.

I had thought the the procedure for testing a new/proposed library would be:

a) clone the whole of boost on one's local machine b) build b2 according to the "bootstrap" instructions c) paste in my new/proposed library d) run b2 headers d) build all the libraries according to the "Getting Started" instructions. This might not build my pasted in library which is fine e) cd to mylibrary/test and run b2 toolset=... etc (or in my case run library status...)

Then one should have one's library built and tested.

Hmm I admit I haven't actually done exactly this. It just never occurred to me that it wouldn't work. Can I assume that works or not?

It works for me as far as my library's tests and doc, so I really don't understand anyone who think it does not.

...

In addition, I would expect that if I go to boost root and run b2 it would build my new library as well. That is I would hope/assume that b2 would build all the libraries in the lib subdirectory rather than depending on some magic list inside of the the jamfile at the root. I know I could investigate it on my own, but it's easier just to ask here while making my point that this is the way it "should" work.

What do you mean by "build my new library as well" ? Are you positing that your theoretical library is a non-header only library ? In which case how does b2 know which jam file to execute to do that ?

...

So in my world, the procededure is past in my new library, re-run b2 headers, and run the library tests - which build the library if necessary. That seems very reasonable, understandable and easily doable if it's not setup like that already. We absolutely have to have such a facility to support the review process.

Finally, I would like to support the idea that a prospective library could just support CMake. In this case the lack of a Build directory in the library would mean that the global b2 ignores it. The CMake could be run from the local library directory. There would be some other issues but we'll leave that for another time.

Robert Ramey

7:24 p.m.

On 5/30/15 11:15 AM, Edward Diener wrote:

...

On 5/30/2015 1:41 PM, Robert Ramey wrote:

...

...
In addition, I would expect that if I go to boost root and run b2 it would build my new library as well. That is I would hope/assume that b2 would build all the libraries in the lib subdirectory rather than depending on some magic list inside of the the jamfile at the root. I know I could investigate it on my own, but it's easier just to ask here while making my point that this is the way it "should" work.

What do you mean by "build my new library as well" ? Are you positing that your theoretical library is a non-header only library ? In which case how does b2 know which jam file to execute to do that ?

I would expect it to walk the directory structure looking at all the libs/*/build and building the libraries or better yet libs/*/test and invoking all the jamfiles it's finds in those directories. If boost build doesn't do it this way but rather depends upon some list, well it would be easy for boost build to generate the list from the directory structure - no other changes necessary. Actually, this would be quite easy for me to implement as as shell script. I realize that this would repeat some dependency checking but it would still work. In general, the building of all of boost should be the union of building each library in the libs directory. Similar for test. Robert Ramey

Steven Watanabe

8:37 p.m.

AMDG On 05/30/2015 01:24 PM, Robert Ramey wrote:

...

If boost build doesn't do it this way but rather depends upon some list, well it would be easy for boost build to generate the list from the directory structure - no other changes necessary.

Actually, this would be quite easy for me to implement as as shell script. I realize that this would repeat some dependency checking but it would still work. In general, the building of all of boost should be the union of building each library in the libs directory. Similar for test.

Building is currently done this way (with a few special cases). The tests have a hard-coded list in status/. In Christ, Steven Watanabe

Rene Rivera

31 May 31 May

4:20 a.m.

On Sat, May 30, 2015 at 3:37 PM, Steven Watanabe wrote:

...

AMDG

On 05/30/2015 01:24 PM, Robert Ramey wrote:

...
If boost build doesn't do it this way but rather depends upon some list, well it would be easy for boost build to generate the list from the directory structure - no other changes necessary.

Actually, this would be quite easy for me to implement as as shell script. I realize that this would repeat some dependency checking but it would still work. In general, the building of all of boost should be the union of building each library in the libs directory. Similar for test.

Building is currently done this way (with a few special cases). The tests have a hard-coded list in status/.

I've wished for some time that was not the case. And that we could do a simple glob and automate the set of tested libraries. But the non-flat structure of the current arrangement makes that much harder than just having a manual list. I've mentioned before that I would very much prefer if we didn't have libraries within libraries in the libs structure. As a flat structure would make it possible to automate. But library authors have ignored my view on this. Note this also make the root build files more complicated than they need to be. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

Edward Diener

4:48 a.m.

On 5/31/2015 12:20 AM, Rene Rivera wrote:

...

On Sat, May 30, 2015 at 3:37 PM, Steven Watanabe wrote:

...
AMDG

On 05/30/2015 01:24 PM, Robert Ramey wrote:

...
If boost build doesn't do it this way but rather depends upon some list, well it would be easy for boost build to generate the list from the directory structure - no other changes necessary.

Actually, this would be quite easy for me to implement as as shell script. I realize that this would repeat some dependency checking but it would still work. In general, the building of all of boost should be the union of building each library in the libs directory. Similar for test.

Building is currently done this way (with a few special cases). The tests have a hard-coded list in status/.

I've wished for some time that was not the case. And that we could do a simple glob and automate the set of tested libraries. But the non-flat structure of the current arrangement makes that much harder than just having a manual list. I've mentioned before that I would very much prefer if we didn't have libraries within libraries in the libs structure. As a flat structure would make it possible to automate. But library authors have ignored my view on this. Note this also make the root build files more complicated than they need to be.

This is not a problem if there were an agreement as to the directory structure of a Boost library in the directory tree. But aside from the 'include' directory structure so that symlinks and 'b2 headers' can be set up to work correctly I don't believe there is such an agreement. I do not believe that a flat directory structure is optimal. There are libraries that should be nested within other libraries if conceptually this is the case.

Rene Rivera

4:59 a.m.

On Sat, May 30, 2015 at 11:48 PM, Edward Diener wrote:

...

On 5/31/2015 12:20 AM, Rene Rivera wrote:

...
On Sat, May 30, 2015 at 3:37 PM, Steven Watanabe wrote:

AMDG

...
On 05/30/2015 01:24 PM, Robert Ramey wrote:

...
If boost build doesn't do it this way but rather depends upon some list, well it would be easy for boost build to generate the list from the directory structure - no other changes necessary.

Actually, this would be quite easy for me to implement as as shell script. I realize that this would repeat some dependency checking but it would still work. In general, the building of all of boost should be the union of building each library in the libs directory. Similar for test.

Building is currently done this way (with a few special cases). The tests have a hard-coded list in status/.

I've wished for some time that was not the case. And that we could do a simple glob and automate the set of tested libraries. But the non-flat structure of the current arrangement makes that much harder than just having a manual list. I've mentioned before that I would very much prefer if we didn't have libraries within libraries in the libs structure. As a flat structure would make it possible to automate. But library authors have ignored my view on this. Note this also make the root build files more complicated than they need to be.

This is not a problem if there were an agreement as to the directory structure of a Boost library in the directory tree. But aside from the 'include' directory structure so that symlinks and 'b2 headers' can be set up to work correctly I don't believe there is such an agreement.

I'm not sure if I understand your assertion.. As we've had this < http://www.boost.org/development/requirements.html#Directory_structure> for, IIRC, at least a decade.

...

I do not believe that a flat directory structure is optimal. There are libraries that should be nested within other libraries if conceptually this is the case.

Please explain in detail and with examples. I truly do want to know why you think this. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

Edward Diener

5:27 a.m.

On 5/31/2015 12:59 AM, Rene Rivera wrote:

...

On Sat, May 30, 2015 at 11:48 PM, Edward Diener wrote:

...
On 5/31/2015 12:20 AM, Rene Rivera wrote:

...
On Sat, May 30, 2015 at 3:37 PM, Steven Watanabe wrote:

AMDG

...
On 05/30/2015 01:24 PM, Robert Ramey wrote:

...
If boost build doesn't do it this way but rather depends upon some list, well it would be easy for boost build to generate the list from the directory structure - no other changes necessary.

Actually, this would be quite easy for me to implement as as shell script. I realize that this would repeat some dependency checking but it would still work. In general, the building of all of boost should be the union of building each library in the libs directory. Similar for test.

Building is currently done this way (with a few special cases). The tests have a hard-coded list in status/.

I've wished for some time that was not the case. And that we could do a simple glob and automate the set of tested libraries. But the non-flat structure of the current arrangement makes that much harder than just having a manual list. I've mentioned before that I would very much prefer if we didn't have libraries within libraries in the libs structure. As a flat structure would make it possible to automate. But library authors have ignored my view on this. Note this also make the root build files more complicated than they need to be.

This is not a problem if there were an agreement as to the directory structure of a Boost library in the directory tree. But aside from the 'include' directory structure so that symlinks and 'b2 headers' can be set up to work correctly I don't believe there is such an agreement.

I'm not sure if I understand your assertion.. As we've had this < http://www.boost.org/development/requirements.html#Directory_structure> for, IIRC, at least a decade.

You are right and I am wrong. But then I do not see why you think a non-flat structure presents problems automating the set of tested libraries ?

...

...
I do not believe that a flat directory structure is optimal. There are libraries that should be nested within other libraries if conceptually this is the case.

Please explain in detail and with examples. I truly do want to know why you think this.

Because certain libraries are conceptually part of a larger concept, and I think those libraries being nested within either another library, or a directory representing the larger concept, is a natural way to represent them within a directory tree. Forcing the directory tree to be flat does not seem to me to be necessary. Let's say with the sort library we have a number of different sort algorithm in the future and each of those algorithms has its own library of functionality. Why not allow them to be under a general 'sort' directory for clarity. Using the file directory structure as a grouping mechanism seems natural to me.

Peter Dimov

10:22 a.m.

Rene Rivera wrote:

...

I've mentioned before that I would very much prefer if we didn't have libraries within libraries in the libs structure.

I've mentioned before that I agree, but will say so again.

Bjørn Roald

1:58 p.m.

On 31. mai 2015 12:22, Peter Dimov wrote:

...

Rene Rivera wrote:

...
I've mentioned before that I would very much prefer if we didn't have libraries within libraries in the libs structure.

I've mentioned before that I agree, but will say so again.

Is it really a question of libraries within libraries in the boost library sense where a library is maintained by individuals or a group of individuals. Or are this just allowing structure within a library, where subdirectories with its own standard structure may for instance be optional parts of the library which when used add additional dependencies. One real challenge with requiring such structures out in separate libs/foo directories is that they then will have to live in separate git repositories, the only way around that is to change the standard structure some other way to allow single top level directory for each git submodule or boost library each having multiple standard directory structures. -- Bjørn

Peter Dimov

6:28 p.m.

Bjørn Roald wrote:

...

On 31. mai 2015 12:22, Peter Dimov wrote:

...
Rene Rivera wrote:

...
I've mentioned before that I would very much prefer if we didn't have libraries within libraries in the libs structure.

I've mentioned before that I agree, but will say so again.

Is it really a question of libraries within libraries in the boost library sense where a library is maintained by individuals or a group of individuals. Or are this just allowing structure within a library, where subdirectories with its own standard structure may for instance be optional parts of the library which when used add additional dependencies.

It is basically a question of removing the current special case of libs/numeric/* and not introducing any others. The libraries in numeric/ are already in their own separate repositories.

Andrey Semashev

7:03 p.m.

On Sunday 31 May 2015 21:28:54 Peter Dimov wrote:

...

Bjørn Roald wrote:

...
On 31. mai 2015 12:22, Peter Dimov wrote:

...
Rene Rivera wrote:

...
I've mentioned before that I would very much prefer if we didn't have libraries within libraries in the libs structure.

I've mentioned before that I agree, but will say so again.

Is it really a question of libraries within libraries in the boost library sense where a library is maintained by individuals or a group of individuals. Or are this just allowing structure within a library, where subdirectories with its own standard structure may for instance be optional parts of the library which when used add additional dependencies.

It is basically a question of removing the current special case of libs/numeric/* and not introducing any others. The libraries in numeric/ are already in their own separate repositories.

Components under tools/ can also be viewed as separate components that can be modularized. In general, I don't think there should be a restriction on the components hierarchy. Forcing libraries to fit into flat structure under libs/ is an artificial restriction imposed by the build/testing system, IMHO.

Peter Dimov

9:09 p.m.

Andrey Semashev wrote:

...

Forcing libraries to fit into flat structure under libs/ is an artificial restriction imposed by the build/testing system, IMHO.

Every restriction is artificial, because Boosts don't exist in nature. I happen to agree with the cost/benefit analysis on this one.

Andrey Semashev

9:27 p.m.

On Monday 01 June 2015 00:09:45 Peter Dimov wrote:

...

Andrey Semashev wrote:

...
Forcing libraries to fit into flat structure under libs/ is an artificial restriction imposed by the build/testing system, IMHO.

Every restriction is artificial, because Boosts don't exist in nature.

What I meant is that I see no problem with searching for test directories, like it is done when building Boost now. status/Jamfile.v2 should not exist in modular Boost, just as the central expected failures markup.

Bjørn Roald

7:21 p.m.

On 31. mai 2015 20:28, Peter Dimov wrote:

...

Bjørn Roald wrote:

...
On 31. mai 2015 12:22, Peter Dimov wrote:

...
Rene Rivera wrote:

...
I've mentioned before that I would very much prefer if we didn't have >> libraries within libraries in the libs structure.

I've mentioned before that I agree, but will say so again.

Is it really a question of libraries within libraries in the boost library sense where a library is maintained by individuals or a group of individuals. Or are this just allowing structure within a library, where subdirectories with its own standard structure may for instance be optional parts of the library which when used add additional dependencies.

It is basically a question of removing the current special case of libs/numeric/* and not introducing any others. The libraries in numeric/ are already in their own separate repositories.

sure, but among other effects such a restriction will have is that further modularization likely will require a lot of tiny git axillary repositories to handle optional glue between boost libraries or glue to optional external technologies. That is if not an alternative, perhaps superior, method to manage such optional dependencies can be realized. I sort of dislike the extremely hard tie between a git repository and the physical layout of directories in it. Some standard structure is clearly needed, but allowing this to recurse does not seem too hard or very unreasonable. If someone want to have more structure within their repository, that ought to be allowed. Detecting subdirectories with the conventional directory structure is one posibility, another is to require such subdirs to be declared explicitly when needed, which is very common in many build systems. Just my 0.05$ anyways. -- Bjørn

Peter Dimov

30 May 30 May

10:29 a.m.

Robert Ramey wrote:

...

Why would anyone want to use this? and for what?

It's a Linux thing. On Linux, you do apt-get install boost and you get a pre-built Boost release (1.55.0, for example, for the current Debian/Ubuntu distributions) automatically downloaded and installed into the system header and library locations. So if you then want to upgrade just Boost.Python, you could in theory download a standalone Boost.Python release and build it with the system-installed Boost.Build, against the system-installed dependencies such as Boost.SmartPtr. On Windows (and, I suppose, OS X), there's no such system-supplied Boost, so the above doesn't apply and Windows people don't see a point, in a similar manner to how Linux people can't see the point of bpm.

Robert Ramey

6:18 p.m.

On 5/30/15 3:29 AM, Peter Dimov wrote:

...

Robert Ramey wrote:

...
Why would anyone want to use this? and for what?

It's a Linux thing.

On Linux, you do

apt-get install boost

and you get a pre-built Boost release (1.55.0, for example, for the current Debian/Ubuntu distributions) automatically downloaded and installed into the system header and library locations.

So if you then want to upgrade just Boost.Python, you could in theory download a standalone Boost.Python release and build it with the system-installed Boost.Build, against the system-installed dependencies such as Boost.SmartPtr.

On Windows (and, I suppose, OS X), there's no such system-supplied Boost, so the above doesn't apply and Windows people don't see a point, in a similar manner to how Linux people can't see the point of bpm.

OK - this is making some more sense to me now. If one uses apt-get to install the total boost package, it installs in the system libraries. So if one want's to paste his own library or updated library into the system, you have to muck with stuff you don't want to. In creating the Boost Library Incubator I wanted to make it easy to download/clone a library and run it with the current boost implementation. After some experimentation on my OSX system I came to recommend the following build/test with CMake/CTest to a directory outside the boost directory. The CMake script uses Find to find the boost installation and set up the include and other paths. This avoids the whole bin.b2 directory structure so it's a simpler albeit less powerful than Boost Build. But one does endup with a library structure which works with boost and is only missing bjam files for build and test in order to fit in the official boost structure. Soo ... it seems that what would be interesting would be the facility to use boost build outside the boost structure - something we don't have yet. This would permit building/testing adding a new library or overriding the downloaded one with a newer version. Have I got this right? Robert Ramey

Rene Rivera

31 May 31 May

4:53 a.m.

On Sat, May 30, 2015 at 5:29 AM, Peter Dimov wrote:

...

Robert Ramey wrote:

Why would anyone want to use this? and for what?

...
It's a Linux thing.

On Linux, you do

apt-get install boost

and you get a pre-built Boost release (1.55.0, for example, for the current Debian/Ubuntu distributions) automatically downloaded and installed into the system header and library locations.

So if you then want to upgrade just Boost.Python, you could in theory download a standalone Boost.Python release and build it with the system-installed Boost.Build, against the system-installed dependencies such as Boost.SmartPtr.

On Windows (and, I suppose, OS X), there's no such system-supplied Boost, so the above doesn't apply and Windows people don't see a point, in a similar manner to how Linux people can't see the point of bpm.

On Windows there's NuGet https://www.nuget.org/packages/boost/. And on OSX there are various package manages (homebrew, macports, fink) and Xcode integration packaging equivalent to NuGet (cocoapods). -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

Niall Douglas

30 May 30 May

12:58 p.m.

On 29 May 2015 at 20:11, Robert Ramey wrote:

...

...
as I have mentioned here before, I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost.

Why would anyone want to use this? and for what?

Specifically for Boost.Python, a very, very long time ago it had been thought it more appropriate that a C++ bindings generator for Python really ought to be part of the Python distribution, not outside it. It would surely get a lot more love and attention being part of official Python and part of their release and unit testing. However back in 2001-2002 Boost.Python needed enough of Boost it was too much of a pain for the Python maintainers. Now we have the C++ 11 STL, I see no reason why Boost.Python needs any of Boost at all anymore as everything it uses in Boost can be substituted with the C++ 11 STL. That should allow Python to adopt Boost.Python into itself, and they take over maintenance and keep both Python and Boost.Python synchronised. If Stefan were so motivated, I even think money could be found from the Boost.Python community to pay for that conversion given the enormous long term benefits it would have to anyone wanting to use C++ with Python and not deal with SWIG. And I think Boost in that situation should willingly and gladly give up Boost.Python if Python core were to adopt the code base instead. Boost.Python should always have been in the core Python distribution. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

David Sankel

3:19 a.m.

On Fri, May 29, 2015 at 9:31 AM, Stefan Seefeld wrote:

...

I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost. I would like to separate the build and release processes. I would like to know whether people here (and Boost.Python users in particular) have any particular thoughts about this proposal. Any strong reasons not to move forward with this ? Things to consider while doing it ? Etc.

First off, thanks for your willingness to put the time and effort into spearheading a continuation of the modularization of boost. It's that kind of volunteerism that the greatness of Boost really rests upon. My hat's off to you. We use Boost.Python and the rest of Boost heavily at Stellar Science. Software approvals might be a bit of a pain with this change, but I don't see a way to avoid that with the current direction of modularization. The big thing I would need to see in order to support your proposal is an explanation of how it would handle dependency issues. Which version of the rest of Boost would Boost.Python require? Is there any chance that a Boost upgrade would break the current release of Boost.Python? If other libraries follow suit, how would the dependency issues effect them? As a user of Boost, what extra steps or effort would be needed to deal with the new dependency problem? (One benefit of the current lock-step release model is that we get an implicit guarantee that all the boost libraries in a particular release are compatible with each other). Thanks in advance for your thoughts. -- David Sankel

Stefan Seefeld

3:55 a.m.

On 29/05/15 11:19 PM, David Sankel wrote:

...

The big thing I would need to see in order to support your proposal is an explanation of how it would handle dependency issues.

That's a good question. Ideally, Boost would be componentized to the point where it would be easy to enumerate Boost.Python's dependencies not on header files but on projects. But that's somewhat tangential to the actual issues, so let's for now assume a simple split between Boost.Python and the rest (or the subset of Boost that Boost.Python requires).

...

Which version of the rest of Boost would Boost.Python require?

Ideally it wouldn't matter. In practice, there is of course the danger that some change would be introduced to Boost (the remainder) that breaks Boost.Python. This is a problem, well beyond Boost.Python, as this means that such a change may affect other Boost users, too. Unfortunately, so far there is absolutely no guarantee that any Boost release N+1 is backward compatible with release N. I believe that it is more than time for Boost to strive for backward compatibility, and to make it very clear whenever any API-level incompatibilities are introduced, so users can adjust. With that knowledge, Boost.Python can address such changes using the usual logic (configure / compile-time checks with conditional code). As I mentioned again and again, such a process is extremely common in the Free Software world when projects rely on third-party software (mostly libraries), and need to adjust to API changes.

...

Is there any chance that a Boost upgrade would break the current release of Boost.Python?

Sure, but I think it's Boost's (the remainder's) responsibility to document whenever such a change is introduced, to allow Boost.Python developer's (as well as any Boost users who may encounter such a change, for that matter !) to adjust.

...

If other libraries follow suit, how would the dependency issues effect them? As a user of Boost, what extra steps or effort would be needed to deal with the new dependency problem? (One benefit of the current lock-step release model is that we get an implicit guarantee that all the boost libraries in a particular release are compatible with each other).

Yes, and I agree that is a big convenience. Whenever a project 'A' depends on libraries 'B' and 'C' (say), it needs to run tests to determine that a particular range of versions of 'B' and 'C' work with 'A' (whether that requires any conditional logic in 'A' or not), and document that properly (as well as encode that into the build system to flag attempts to build against incompatible versions of 'B' and 'C'). Again, the real issue here is quite independent of my attempt to decouple Boost.Python from the rest of Boost: It's that as far as Boost is concerned, there has never been any explicit or implied guarantee that subsequent Boost releases are API compatible (or even ABI compatible !). I think it's time to rethink that.

...

Thanks in advance for your thoughts.

Thanks for your encouraging feedback ! Stefan -- ...ich hab' noch einen Koffer in Berlin...

Peter Dimov

10:37 a.m.

Stefan Seefeld wrote:

...

Ideally, Boost would be componentized to the point where it would be easy to enumerate Boost.Python's dependencies not on header files but on projects.

Boost is already componentized to such a point. Read the dependency report. It lists the libraries on which Boost.Python depends.

...

Unfortunately, so far there is absolutely no guarantee that any Boost release N+1 is backward compatible with release N. I believe that it is more than time for Boost to strive for backward compatibility, and to make it very clear whenever any API-level incompatibilities are introduced, so users can adjust.

The way we currently guarantee that Boost release N+1 is backward compatible with Boost release N with respect to Boost.Python is by running the tests of Boost.Python against other libraries in release N+1 before release. If you pull Boost.Python out of the Boost release procedure, nobody is going to run the tests of Boost.Python against N+1 before N+1 is released, and regressions will not be caught. This is not the first time I'm saying this, I believe. There must be some sort of a misunderstanding.

Sergey Popov

4:25 p.m.

Although supporting standalone builds of libraries against already installed boost would be really nice I consider going to purely separate library releases to be a horrible idea. I never thought that monolithic nature of boost is problematic. More like it makes everyone's lives easier because it's simpler to reason about versions for library users. They don't need to track library version requirements for individual libraries which can get really tricky if those libraries depend on more libraries. Instead, depending on particular version of boost is enough.

Rene Rivera

31 May 31 May

4:14 a.m.

On Fri, May 29, 2015 at 11:31 AM, Stefan Seefeld wrote:

...

as I have mentioned here before, I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost. For avoidance of doubt: my intent is not to remove Boost.Python out of the Boost organization. Rather, I would like to separate the build and release processes. For the sake of keeping the discussion focused, I would like to avoid getting into too many (technical) details (there already is a related discussion on the Boost.Build list). Rather, I would like to know whether people here (and Boost.Python users in particular) have any particular thoughts about this proposal. Any strong reasons not to move forward with this ? Things to consider while doing it ? Etc.

(My hope is that if the operation succeeds, other Boost libraries may want to do the same, which would help Boost in the long term to become more manageable.)

In summary.. As the author of the only Boost library that is both independently modular and at the same time part of Boost monolithic I'm not entirely opposed to the concept of making libraries modular. I do have reservations, as others have mentioned, with removing libraries from the monolithic release. I prefer the route of allowing the modular use of the libraries but keeping the "comprehensive" release (note my avoidance of monolithic). In detail.. And with the various hats I wear in Boost.. As a library author I would like to: a) Continue the development of my library as an independent entity as it makes development and testing easier and more predictable. b) Continue to take advantage of the distribution of my library as part of the "comprehensive" Boost release. In my opinion such distribution hoists the library into a position of being more likely to be used than an independent decoupled release. c) I would like to allow users of the library the option of using updated versions of the library independent of the "comprehensive" Boost release cycle. d) I would like to continue to allow users the option of independently using my library. e) I would like to continue to have my library tested as part of an integrated and quality tested Boost release. f) I would like to have continuous testing of my library in as many platforms as possible. As a user of multiple Boost libraries I would like to: a) Continue to have the option to recommend and get clearance on a comprehensive Boost release for projects. As it is some times easier to get such approval for the whole as opposed to the parts. b) Continue to have the ability to recommend, get clearance, and use individual modular Boost libraries as needed for individual projects. As it is some times easier to get such approval on a smaller subset of libraries than the whole of Boost. Note, I said continue because I already do this with one of my projects with the "modular.jam" Boost Build extension. As Testing Manager, and Release Team member, I would like to: a) Continue to provide a comprehensive set of testing results for all accepted Boost libraries. So that I can monitor the release state of the comprehensive Boost release. b) I would like to extend testing to include cloud/distributed continuous testing of all accepted Boost libraries. And this aspect is really only possible by testing modular libraries instead of a monolithic release because of resource constraints (i.e. as Travis-CI and others are time limited resources). c) I would like to have a fully integrated comprehensive and packaged Boost release continuously tested. So that doing Boost releases is a low cost endeavor. Definition.. What is this "comprehensive" Boost release that I keep mentioning? It's a release that packages a fully tested set of modular Boost libraries that has been tested to guarantee (within the obvious uncertainties) correct functionality of those libraries and their dependencies. But it is not a release that requires the use of the libraries as a monolithic single package. In conclusion.. I have the feeling that some of your goals intersect with mine. But I also suspect some of your goals will collide with mine. But I thought I'd mention what my position is since you asked for feedback on your efforts. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

Niall Douglas

10:45 a.m.

On 30 May 2015 at 23:14, Rene Rivera wrote:

...

As the author of the only Boost library that is both independently modular and at the same time part of Boost monolithic I'm not entirely opposed to the concept of making libraries modular.

I didn't know that about Predef. Boost.Build is also independently modular and part of Boost monolithic, though true it is not a library. Boost.ASIO is generated by automated scripts from standalone ASIO. If Predef were available as a single giant include file, I think it would be a lot more useful to non-Boost code. I'm specifically thinking of APIBind. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

Rob Stewart

11:59 a.m.

On May 31, 2015 12:14:57 AM EDT, Rene Rivera wrote:

...

On Fri, May 29, 2015 at 11:31 AM, Stefan Seefeld wrote:

...
as I have mentioned here before, I would very much like to continue the modularization effort, by fully separating the Boost.Python project from the rest of Boost. For avoidance of doubt: my intent is not to remove Boost.Python out of the Boost organization. Rather, I would like to separate the build and release processes.

As the author of the only Boost library that is both independently modular and at the same time part of Boost monolithic I'm not entirely opposed to the concept of making libraries modular. I do have reservations, as others have mentioned, with removing libraries from the monolithic release. I prefer the route of allowing the modular use of the libraries but keeping the "comprehensive" release (note my avoidance of monolithic).

[snip details] You've captured, in detail, the idea I tried to convey: modular is very likely a good idea, at least for many libraries, but a monolithic/comprehensive release remains necessary. ___ Rob (Sent from my portable computation engine)

3468

Age (days ago)

3470

Last active (days ago)

List overview

Download

62 comments

18 participants

participants (18)

Andrey Semashev
Bjorn Reese
Bjørn Roald
David Sankel
Edward Diener
Nevin Liber
Niall Douglas
Oleg Grunin
Pete Bartlett
Peter Dimov
Rene Rivera
Rob Stewart
Robert Ramey
Sergey Cheban
Sergey Popov
Stefan Seefeld
Steven Watanabe
Vladimir Prus