On 04/06/2015 09:00 AM, Robert Ramey wrote:

Yipes

Looks like accidentally pushed to the wrong branch on March 21 - 22. I honestly don't remember how I did this. One can either role back serialization to March 4 (cf0d7de) or I can merge in the develop branch which is currently passing all tests.

I have reverted 4 commits to master made since cf0d7de and updated superproject reference. Olaf, could you give current master a try? Robert, when you do next merge to master, please examine results carefully. I did revert using 'git revert', and 'git merge' might not notice that, so you might have to use 'git cherry-pick' to bring back those 4 commits to master. HTH, -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com

On Mon, Apr 6, 2015 at 2:45 PM, Beman Dawes wrote:

On Sun, Apr 5, 2015 at 5:51 PM, Marshall Clow wrote:

...

Release candidate files for 1.58.0 are available at: http://boost.cowic.de/rc/

I downloaded boost_1_58_0_rc1.7z, checked the MD5, and built it (Win7 64-bit, msvc-12.0)

Did you do a 32-bit or a 64-bit build?

The build was clean - no errors reported.

Since this was the same setup that failed for Olaf, I have no idea why it worked for me. I looked at archive_exception.cpp in the log and it definitely built OK.

b2 --build-type=complete ? That's not the default unfortunatrely. -- Olaf

On Mon, Apr 6, 2015 at 9:01 AM, Beman Dawes wrote:

On Mon, Apr 6, 2015 at 8:51 AM, Olaf van der Spek wrote:

...

...

The build was clean - no errors reported.

Since this was the same setup that failed for Olaf, I have no idea why it worked for me. I looked at archive_exception.cpp in the log and it definitely built OK.

b2 --build-type=complete ?

On Mon, Apr 6, 2015 at 2:45 PM, Beman Dawes wrote:

...

...

On Sun, Apr 5, 2015 at 5:51 PM, Marshall Clow wrote:

...

Release candidate files for 1.58.0 are available at: http://boost.cowic.de/rc/

I downloaded boost_1_58_0_rc1.7z, checked the MD5, and built it (Win7 64-bit, msvc-12.0)

No. I'm rerunning now with --build-type=complete

Failed, with the same errors you reported. That's a cautionary tale! Marshall, We need to start asking RC testers to use --build-type=complete --Beman

On Mon, Apr 6, 2015 at 10:08 AM, Olaf van der Spek wrote:

On Mon, Apr 6, 2015 at 3:53 PM, Beman Dawes wrote:

...

We need to start asking RC testers to use --build-type=complete

Why doesn't it default to complete (on Windows)?

Good point. Likewise, why isn't the default a 64-bit build on a 64-bit system? There are also some other build defaults and names that may sense in isolation but taken as a whole make much less sense. We need to have a discussion about that early in the 1.59.0 release cycle. --Beman

Beman Dawes wrote:

Likewise, why isn't the default a 64-bit build on a 64-bit system?

Because you may want the software you're developing to run on 32 bit OSes, which is still the most common use case on Windows. Ideally, you need the default on Windows to be to build both, but the address model is not encoded in the name, so that's not possible at present.

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Peter Dimov Sent: 06 April 2015 16:31 To: boost@lists.boost.org Subject: Re: [boost] [1.58.0] Release candidates available

Beman Dawes wrote:

...

Likewise, why isn't the default a 64-bit build on a 64-bit system?

Because you may want the software you're developing to run on 32 bit OSes, which is still the most common use case on Windows.

Ideally, you need the default on Windows to be to build both, but the address model is not encoded in the name, so that's not possible at present.

I think we need to discuss changing that. Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 (0) 1539 561830

Am 30.04.2015 um 13:57 schrieb Olaf van der Spek:

On Mon, Apr 6, 2015 at 4:08 PM, Olaf van der Spek wrote:

...

On Mon, Apr 6, 2015 at 3:53 PM, Beman Dawes wrote:

...

...

No. I'm rerunning now with --build-type=complete

Failed, with the same errors you reported.

That's a cautionary tale!

Marshall,

We need to start asking RC testers to use --build-type=complete

Why doesn't it default to complete (on Windows)? Where's build-type documented anyway?

All I found was " For instructions on how to build only specific variants, please ask on the Boost.Build mailing list."

Somebody?

You can call "b2 --help" (or "bjam --help") in the Boost-source directory and will get the information you are looking for: --build-type=<type> Build the specified pre-defined set of variations of the libraries. Note, that which variants get built depends on what each library supports. -- minimal -- (default) Builds a minimal set of variants. On Windows, these are static multithreaded libraries in debug and release modes, using shared runtime. On Linux, these are static and shared multithreaded libraries in release mode. -- complete -- Build all possible variations. Additionally, you might find some more useful information about boost.build here: http://www.boost.org/build/doc/html/bbv2/overview.html Best regards, Deniz

Am 30.04.2015 um 18:46 schrieb Olaf van der Spek:

On Thu, Apr 30, 2015 at 6:03 PM, Deniz Bahadir wrote:

...

Am 30.04.2015 um 13:57 schrieb Olaf van der Spek:

...

On Mon, Apr 6, 2015 at 4:08 PM, Olaf van der Spek wrote:

...

On Mon, Apr 6, 2015 at 3:53 PM, Beman Dawes wrote:

...

...

No. I'm rerunning now with --build-type=complete

Failed, with the same errors you reported.

That's a cautionary tale!

Marshall,

We need to start asking RC testers to use --build-type=complete

Why doesn't it default to complete (on Windows)? Where's build-type documented anyway?

All I found was " For instructions on how to build only specific variants, please ask on the Boost.Build mailing list."

Somebody?

You can call "b2 --help" (or "bjam --help") in the Boost-source directory and will get the information you are looking for:

--build-type=<type> Build the specified pre-defined set of variations of the libraries. Note, that which variants get built depends on what each library supports. -- minimal -- (default) Builds a minimal set of variants. On Windows, these are static multithreaded libraries in debug and release modes, using shared runtime. On Linux, these are static and shared multithreaded libraries in release mode. -- complete -- Build all possible variations.

Ah, thanks. Could the -s variant (static mt lib release with static runtime) be build too? I think it's a relatively frequently used variant.

As far as I know this should be possible, if you do not use the build-option "--build-type" but instead the build-properties "variant", "link", "threading" and "runtime-link". (See "b2 --help".) For the combination you asked for the build-properties on the command-line would look like: variant=release link=static runtime-link=static threading=multi

I still think the documentation should be improved such that it doesn't have to redirect to a mailing list for further info.

I agree with you. Someone, who feels in charge for boost-build should probably consider doing this. Deniz -- BENOCS GmbH Dipl.-Inform. Deniz Bahadir Winterfeldtstr. 21 10781 Berlin Germany Phone: +49 - 30 / 577 0004-22 Email: deniz.bahadir@benocs.com www.benocs.com Board of Management: Michael Wolz, Dr.-Ing. Oliver Holschke, Dr.-Ing. Ingmar Poese Commercial Register: Amtsgericht Bonn HRB 19378

On Mon, Apr 6, 2015 at 11:07 AM, Paul A. Bristow wrote:

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Beman Dawes Sent: 06 April 2015 14:53 To: Boost Developers List Subject: Re: [boost] [1.58.0] Release candidates available

On Mon, Apr 6, 2015 at 9:01 AM, Beman Dawes wrote:

We need to start asking RC testers to use --build-type=complete

--Beman

Why isn't this the default?

If you search back through email history you'd find that it used to be the default. But was changed as doing complete was: rarely what end users wanted, used up gobs of disk space, and made the build take a really long time (there might be other reasons but I don't remember ATM). -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

On Mon, Apr 6, 2015 at 6:38 PM, Stephan T. Lavavej < stl@exchange.microsoft.com> wrote:

...

If you search back through email history you'd find that it used to be

[Rene Rivera] the

...

default. But was changed as doing complete was: rarely what end users wanted, used up gobs of disk space, and made the build take a really long time (there might be other reasons but I don't remember ATM).

On my 3-year-old dev box with a spinny hard drive, extracting Boost 1.57.0 takes 4 minutes, and a complete x86 build plus cleanup takes 10 minutes, resulting in 2.03 GB of headers and DLLs/LIBs. Neither seems especially onerous, aside from the fact that the Getting Started documentation doesn't say to use a parallel build, which is *really* bad.

STL

Basically the same experience for me. Although I didn't know that there was a parallel build option.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On 7/04/2015 4:39 AM, "Stephan T. Lavavej" wrote:

[Rene Rivera]

...

If you search back through email history you'd find that it used to be

the

...

default. But was changed as doing complete was: rarely what end users wanted, used up gobs of disk space, and made the build take a really long time (there might be other reasons but I don't remember ATM).

On my 3-year-old dev box with a spinny hard drive, extracting Boost 1.57.0 takes 4 minutes, and a complete x86 build plus cleanup takes 10 minutes, resulting in 2.03 GB of headers and DLLs/LIBs. Neither seems especially onerous, aside from the fact that the Getting Started documentation doesn't say to use a parallel build, which is *really* bad.

That's different to running regression tests. Regression tests that I've run take somewhere around 4 hours ;-) And that's with a parallel build.

On Tue, Apr 7, 2015 at 10:45 AM, Olaf van der Spek wrote:

On Mon, Apr 6, 2015 at 6:38 PM, Stephan T. Lavavej wrote:

...

[Rene Rivera]

...

If you search back through email history you'd find that it used to be the default. But was changed as doing complete was: rarely what end users wanted, used up gobs of disk space, and made the build take a really long time (there might be other reasons but I don't remember ATM).

On my 3-year-old dev box with a spinny hard drive, extracting Boost 1.57.0 takes 4 minutes, and a complete x86 build plus cleanup takes 10 minutes, resulting in 2.03 GB of headers and DLLs/LIBs. Neither seems especially onerous, aside from the fact that the Getting Started documentation doesn't say to use a parallel build, which is *really* bad.

Could b2 default to -jX where X is the number of cores?

This would be unexpected and detrimental if there is not enough RAM.

...

and detrimental if there is not enough RAM. How much RAM is needed?

Depends on the compiler, and which builds just so happen to end up in parallel. Broadly I would support this, though I have noticed certain compilers choking when run in parrellel - Intel's is a prime culprit - once the system ram is exhausted, the system just sits there thrashing the swap file and getting nowhere :( John.

AMDG On 04/07/2015 02:38 AM, Andrey Semashev wrote:

On Tue, Apr 7, 2015 at 11:22 AM, Olaf van der Spek wrote:

...

On Tue, Apr 7, 2015 at 10:00 AM, Andrey Semashev wrote:

...

...

Could b2 default to -jX where X is the number of cores?

This would be unexpected

Why?

AFAIK, other build systems do not build in parallel by default.

Ninja does, I believe. In Christ, Steven Watanabe

BuildInParallel

A boolean value that indicates whether project references are built or cleaned in parallel when Multi-Proc MSBuild is used. The default value is true, which means that projects will be built in parallel if the system has multiple cores or processors.

I don't think this means what you think it means -- as far as I understand this means that individual 'projects' will be built in parallel relative to each other only. Thus, if you only have one project, then by default you do not get a parallel build. As far as I know, there is no way in MSBuild to get a single project to build individual source files in parallel. But I would love to be corrected about that. -- chris

On 8/04/2015 09:33, Chris Glover wrote:

...

BuildInParallel

A boolean value that indicates whether project references are built or cleaned in parallel when Multi-Proc MSBuild is used. The default value is true, which means that projects will be built in parallel if the system has multiple cores or processors.

I don't think this means what you think it means -- as far as I understand this means that individual 'projects' will be built in parallel relative to each other only. Thus, if you only have one project, then by default you do not get a parallel build. As far as I know, there is no way in MSBuild to get a single project to build individual source files in parallel. But I would love to be corrected about that.

Not in MSBUILD, but for example CL supports the /MP option which if specified will tell it to compile separate compilation units in parallel. (The default, however, is not to do that.)

On 8/04/2015 16:13, Bjørn Roald wrote:

On 08. april 2015 01:47, Gavin Lambert wrote:

...

Not in MSBUILD, but for example CL supports the /MP option which if specified will tell it to compile separate compilation units in parallel. (The default, however, is not to do that.)

Then you have to list more than one source file for each cl invocation, which is fast, possibly for other reasons. Ant actually did that many years ago when I used it for some C++ work. I do not remember if there was a flag for parallel build added by Ant.

I believe MSBUILD runs CL exactly twice for a normal project -- once to build the precompiled headers and once for all the other files. Obviously this will be only once if you're not using precompiled headers, or possibly many times if you have individual files with different compiler settings. But either way specifying /MP should give you the most theoretical benefit, unless you have lots of projects containing only a few files (and that don't depend on each other), in which case the MSBUILD setting may be more beneficial. I'm not sure what B2 does.

On 04/07/2015 11:40 AM, Rob Stewart wrote:

On April 7, 2015 4:22:05 AM EDT, Olaf van der Spek wrote:

...

On Tue, Apr 7, 2015 at 10:00 AM, Andrey Semashev wrote:

...

...

Could b2 default to -jX where X is the number of cores?

This would be unexpected

Why?

It would be unlike other build tools, like make. It would be an breaking interface change.

...

...

and detrimental if there is not enough RAM.

How much RAM is needed?

It scales almost linearly, I imagine. Each compiler instance requires memory to build a translation unit.

If Boost used to build fine on a limited machine, it might slow to a crawl due to paging with the added parallelism. Such users would have to use -j1 to avoid that problem.

Those with memory enough and more CPUs need to opt in to benefit, of course, but the build doesn't swamp such machines by default. However, if the option could be set in user.jam, then those that wish to opt in can do so once and then forget about it.

Rob, it used to be possible to do: import option ; option.set jobs : 4 ; in user-config.jam, but that appears to have regressed. Is that what you're after? -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com

On 07. april 2015 12:05, Vladimir Prus wrote:

On 04/07/2015 11:40 AM, Rob Stewart wrote:

...

On April 7, 2015 4:22:05 AM EDT, Olaf van der Spek wrote:

...

On Tue, Apr 7, 2015 at 10:00 AM, Andrey Semashev wrote:

...

...

Could b2 default to -jX where X is the number of cores?

This would be unexpected

Why?

It would be unlike other build tools, like make. It would be an breaking interface change.

...

...

and detrimental if there is not enough RAM.

How much RAM is needed?

It scales almost linearly, I imagine. Each compiler instance requires memory to build a translation unit.

If Boost used to build fine on a limited machine, it might slow to a crawl due to paging with the added parallelism. Such users would have to use -j1 to avoid that problem.

Those with memory enough and more CPUs need to opt in to benefit, of course, but the build doesn't swamp such machines by default. However, if the option could be set in user.jam, then those that wish to opt in can do so once and then forget about it.

Rob, it used to be possible to do:

import option ; option.set jobs : 4 ;

in user-config.jam, but that appears to have regressed. Is that what you're after?

This would be a nice config setting. Also, in principle, bjam could check available free RAM before invoking a new parallel task. I think -j 4 does not mean it _has_ to run 4 tasks in parallel, rather it means up to 4 tasks in paralell. An implicit -j certainly should throttle on system resources, so why not available RAM as well as available cores. http://stackoverflow.com/questions/150355/programmatically-find-the-number-o... http://stackoverflow.com/questions/2513505/how-to-get-available-memory-c-g -- Bjørn

On 07. april 2015 22:33, Vladimir Prus wrote:

On 04/07/2015 10:42 PM, Bjørn Roald wrote:

...

Also, in principle, bjam could check available free RAM before invoking a new parallel task. I think -j 4 does not mean it _has_ to run 4 tasks in parallel, rather it means up to 4 tasks in paralell. An implicit -j certainly should throttle on system resources, so why not available RAM as well as available cores.

I don't think it's easy. If I run 4 compilations in parallel, and it consumes so much RAM and I/O that computer becomes unresponsive, it means the OS could not throttle these tasks effectively.

The OS can not throttle a running process' greediness for RAM, at least I don't think so. It could prevent new processes to start, but that is also tricky for the OS in a general sense. This is however trivial for a build system when deciding whether it should start additional parallel compiler invocations that are totally optional tuning for build speed. It make no sense to start additional compilations in parallel if you see the physical RAM is consumed. The tricky part is knowing when to stop adding parallel tasks to prevent getting in a consumed RAM state in the first place. And hopefully still leave ample space for the rest of the system to live. Shooting from the hip here, why not require at least 2G of free RAM to start a task running in parallel. 2G may seem excessively defensive, but it provide good margins and who around here have dev. boxes with less than 8G RAM these days. -- Bjørn

On 08. april 2015 00:12, Niall Douglas wrote:

On 8 Apr 2015 at 0:03, Bjørn Roald wrote:

...

Shooting from the hip here, why not require at least 2G of free RAM to start a task running in parallel. 2G may seem excessively defensive, but it provide good margins and who around here have dev. boxes with less than 8G RAM these days.

That would be an unhelpful heuristic on VMs with balloon allocated memory.

OK, so if I understand that correctly, in such an environment the reported available RAM would be low, but as contention is detected by the balloon driver the guest RAM would be inflated, thus more RAM demanding processes would fit just fine. That is a tricky one. -- Bjørn

On 04/08/2015 01:03 AM, Bjørn Roald wrote:

On 07. april 2015 22:33, Vladimir Prus wrote:

...

On 04/07/2015 10:42 PM, Bjørn Roald wrote:

...

Also, in principle, bjam could check available free RAM before invoking a new parallel task. I think -j 4 does not mean it _has_ to run 4 tasks in parallel, rather it means up to 4 tasks in paralell. An implicit -j certainly should throttle on system resources, so why not available RAM as well as available cores.

I don't think it's easy. If I run 4 compilations in parallel, and it consumes so much RAM and I/O that computer becomes unresponsive, it means the OS could not throttle these tasks effectively.

The OS can not throttle a running process' greediness for RAM, at least I don't think so. It could prevent new processes to start, but that is also tricky for the OS in a general sense. This is however trivial for a build system when deciding whether it should start additional parallel compiler invocations that are totally optional tuning for build speed. It make no sense to start additional compilations in parallel if you see the physical RAM is consumed. The tricky part is knowing when to stop adding parallel tasks to prevent getting in a consumed RAM state in the first place. And hopefully still leave ample space for the rest of the system to live.

On my system right now, there's 142M of free memory (and similar amount of buffers). That does sound sufficient for any compilation these days, still -j4 build is OK, since OS discards most of that memory and swaps the other quite easily. I am not sure we can answer "is there enough RAM" question reliably. Likewise, "will 4 jobs too much I/O" question is not easy to answer. - Volodya -- Vladimir Prus CodeSourcery / Mentor Embedded http://vladimirprus.com

On April 8, 2015 9:29:36 AM EDT, "Paul A. Bristow" wrote:

Would the uber-naïve heuristic -jone_less_than_max_cores work OK-ish (for a desktop machine)?

For a build machine, a good rule of thumb is to use N+1 to account for I/O wait. If you assume a desktop on which a user wants to do other work, then you to back off by one or two. (All of this assumes sufficient memory.) Which to target with the defaults? ___ Rob (Sent from my portable computation engine)

Rene Rivera wrote:

If you search back through email history you'd find that [--build-type=complete] used to be the default. But was changed as doing complete was: rarely what end users wanted, used up gobs of disk space, and made the build take a really long time (there might be other reasons but I don't remember ATM).

This end user always ended up doing the complete build after doing the default build and then discovering that the autolink wants something else. :-)

On Mon, Apr 6, 2015 at 6:11 PM, Rene Rivera wrote:

On Mon, Apr 6, 2015 at 11:07 AM, Paul A. Bristow wrote:

...

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Beman Dawes Sent: 06 April 2015 14:53 To: Boost Developers List Subject: Re: [boost] [1.58.0] Release candidates available

On Mon, Apr 6, 2015 at 9:01 AM, Beman Dawes wrote:

We need to start asking RC testers to use --build-type=complete

--Beman

Why isn't this the default?

If you search back through email history you'd find that it used to be the default. But was changed as doing complete was: rarely what end users wanted, used up gobs of disk space, and made the build take a really long time (there might be other reasons but I don't remember ATM).

Makes sense. Building libraries on demand would probably be the ultimate solution.. Could the default also build the one -s variant? -- Olaf