Docker environment(s) for Boost Development

James E. King, III

8 May 2018 8 May '18

11:59 a.m.

I submitted a pull request into the superproject containing some docker scripts. The intention of this is to simplify managing a docker build environment for boost: https://github.com/boostorg/boost/pull/184 I'm wondering if folks will find this useful, and whether folks think this should be in its own submodule instead of being in boostorg/boost. I only have an Ubuntu Bionic environment in this pull request but it wouldn't be that difficult to add debian, centos, and others, as well as various versions of each. I have tested it on Unix and on Windows. On Unix the performance is quite good overall - but on Windows since they use SMB to do the sharing it is really slow (I would say a good 20 times slower between typing "b2" and getting the first output as it scans through all the files. I would recommend continuing to use a virtual machine on Windows Hosts for unix boost development, but this option is there and hopefully they will improve things. This partially came out of the discussion we had a number of months back about how we handle third party dependencies. For the most part one has to look at README files in various locations to learn about them. This is an attempt to pull all the dependencies together for a complete build into one place (the Dockerfile). One day when Microsoft supports Visual Studio compilation inside a reasonably sized windows container (i.e. based on nanoserver and not Windows Server Core), I'd like to add that as well since managing the dependencies on Windows is a bit harder (you usually have to build them). Thanks, Jim

Show replies by date

Tom Kent

9 May 9 May

11:53 p.m.

On Tue, May 8, 2018 at 6:59 AM, James E. King, III via Boost < boost@lists.boost.org> wrote:

...

I submitted a pull request into the superproject containing some docker scripts. The intention of this is to simplify managing a docker build environment for boost:

https://github.com/boostorg/boost/pull/184

I'm wondering if folks will find this useful, and whether folks think this should be in its own submodule instead of being in boostorg/boost. I only have an Ubuntu Bionic environment in this pull request but it wouldn't be that difficult to add debian, centos, and others, as well as various versions of each.

I have tested it on Unix and on Windows. On Unix the performance is quite good overall - but on Windows since they use SMB to do the sharing it is really slow (I would say a good 20 times slower between typing "b2" and getting the first output as it scans through all the files. I would recommend continuing to use a virtual machine on Windows Hosts for unix boost development, but this option is there and hopefully they will improve things.

This partially came out of the discussion we had a number of months back about how we handle third party dependencies. For the most part one has to look at README files in various locations to learn about them. This is an attempt to pull all the dependencies together for a complete build into one place (the Dockerfile).

One day when Microsoft supports Visual Studio compilation inside a reasonably sized windows container (i.e. based on nanoserver and not Windows Server Core), I'd like to add that as well since managing the dependencies on Windows is a bit harder (you usually have to build them).

Thanks,

Jim

I just wanted to mention that I've got a stable of boost docker images available at: https://hub.docker.com/r/teeks99/boost-cpp-docker/ built from the git repo: https://github.com/teeks99/boost-cpp-docker I initially created these as the source for the linux regression runners that I manage (teeks99-02*, teeks99-03*), but have used them for a lot more. They are more focused on having older versions of compilers (gcc and clang) working, and I think is largely complimentary to what you have here. Just wanted to mention it in this thread. Another thought, it might be a nice idea to take what you've done for the baseline to create an environment, and for each release build the release version of boost inside it and publish it to docker hub. This would give users a way to quickly get a boost development environment, without having to wait for the distros to package (and mangle) the binaries. Tom

Mateusz Loskot

25 Jul 25 Jul

9:17 a.m.

Boost - Dev mailing list wrote

...

On Tue, May 8, 2018 at 6:59 AM, James E. King, III via Boost <

...

boost@.boost

...

...
wrote: I submitted a pull request into the superproject containing some docker scripts. The intention of this is to simplify managing a docker build environment for boost:

https://github.com/boostorg/boost/pull/184

[...] This partially came out of the discussion we had a number of months back about how we handle third party dependencies. For the most part one has to look at README files in various locations to learn about them. This is an attempt to pull all the dependencies together for a complete build into one place (the Dockerfile).

I just wanted to mention that I've got a stable of boost docker images available at: https://hub.docker.com/r/teeks99/boost-cpp-docker/

built from the git repo: https://github.com/teeks99/boost-cpp-docker

I initially created these as the source for the linux regression runners that I manage (teeks99-02*, teeks99-03*)

Tom, Do you also automated running the regression tests? Could you explain your workflow with those docker images you perform to run the complete Boost regression tests suite? Best regards, Mateusz Loskot ----- -- Mateusz Loskot, http://mateusz.loskot.net -- Sent from: http://boost.2283326.n4.nabble.com/Boost-Dev-f2600599.html

Tom Kent

11:36 p.m.

On Wed, Jul 25, 2018 at 4:17 AM, Mateusz Loskot via Boost < boost@lists.boost.org> wrote:

...

Boost - Dev mailing list wrote

...
On Tue, May 8, 2018 at 6:59 AM, James E. King, III via Boost <

...
boost@.boost

...
...
wrote: I submitted a pull request into the superproject containing some docker scripts. The intention of this is to simplify managing a docker build environment for boost:

https://github.com/boostorg/boost/pull/184

[...] This partially came out of the discussion we had a number of months back about how we handle third party dependencies. For the most part one has to look at README files in various locations to learn about them. This is an attempt to pull all the dependencies together for a complete build into one place (the Dockerfile).

I just wanted to mention that I've got a stable of boost docker images available at: https://hub.docker.com/r/teeks99/boost-cpp-docker/

built from the git repo: https://github.com/teeks99/boost-cpp-docker

I initially created these as the source for the linux regression runners that I manage (teeks99-02*, teeks99-03*)

Tom,

Do you also automated running the regression tests?

Could you explain your workflow with those docker images you perform to run the complete Boost regression tests suite?

Best regards, Mateusz Loskot

I do use those images in the regression tests that I run; teeks99-02-*, teeks99-03-* . Those images are setup in dockerhub, so when their upstream images (typically the ubuntu one that was mainstream when that version of the compiler was released) get updates, docker hub will automatically rebuild them with the updates. As far as new versions of the toolset (e.g. when clang drops a .1 build) that is up to me to re-run the images. Which I'm often forgetful about doing. For completely new versions (e.g. gcc7 -> gcc8) I have to manually make up a new config. I didn't mention it above, but I have two separate docker hub repos for keeping the gcc and clang toolsets. These don't have anything boost specific in it (even the boost-cpp-docker doesn't have that much, just some specific dependencies), and I've used them for other purposes. They can be found at: https://github.com/teeks99/gcc-ubuntu-docker https://github.com/teeks99/clang-ubuntu-docker As far as how I use them in the regression tests, I've got a bit of tooling setup here: https://github.com/teeks99/boost-build/tree/master/Regression It is mostly just a couple python scripts that continuously cycle between configurations and the lists of configs that I want to run on each machine, windows and linux. Tom

Mateusz Loskot

26 Jul 26 Jul

5:12 p.m.

On 26 July 2018 at 01:36, Tom Kent via Boost wrote:

...

On Wed, Jul 25, 2018 at 4:17 AM, Mateusz Loskot via Boost wrote:

...
Boost - Dev mailing list wrote

...
On Tue, May 8, 2018 at 6:59 AM, James E. King, III via Boost wrote:

...
I submitted a pull request into the superproject containing some docker scripts. The intention of this is to simplify managing a docker build environment for boost:

https://github.com/boostorg/boost/pull/184

[...] This partially came out of the discussion we had a number of months back about how we handle third party dependencies. For the most part one has to look at README files in various locations to learn about them. This is an attempt to pull all the dependencies together for a complete build into one place (the Dockerfile).

I just wanted to mention that I've got a stable of boost docker images available at: https://hub.docker.com/r/teeks99/boost-cpp-docker/

built from the git repo: https://github.com/teeks99/boost-cpp-docker

I initially created these as the source for the linux regression runners that I manage (teeks99-02*, teeks99-03*)

Tom,

Do you also automated running the regression tests?

Could you explain your workflow with those docker images you perform to run the complete Boost regression tests suite?

[...] As far as how I use them in the regression tests, I've got a bit of tooling setup here: https://github.com/teeks99/boost-build/tree/master/Regression

It is mostly just a couple python scripts that continuously cycle between configurations and the lists of configs that I want to run on each machine, windows and linux.

This looks like something I've been looking for/thinking of. I'm going to try it out in action soonish. Thanks for sharing. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Mateusz Loskot

25 Jul 25 Jul

9:21 a.m.

Boost - Dev mailing list wrote

...

The intention of this is to simplify managing a docker build environment for boost [...] This partially came out of the discussion we had a number of months back about how we handle third party dependencies. For the most part one has to look at README files in various locations to learn about them. This is an attempt to pull all the dependencies together for a complete build into one place (the Dockerfile).

James, TBH, to me, this gives quite a little idea what is the actual purpose and target audience of the docker images. Is it for Boost users, to provide them with Boost itself? Is it for Boost contributors, to provide them with development and testing environment? What do you mean by "all the dependencies", Boost itself or deps required by Boost? Best regards, Mateusz Loskot ----- -- Mateusz Loskot, http://mateusz.loskot.net -- Sent from: http://boost.2283326.n4.nabble.com/Boost-Dev-f2600599.html

James E. King III

10:50 a.m.

On Wed, Jul 25, 2018 at 5:41 AM Mateusz Loskot via Boost < boost@lists.boost.org> wrote:

...

Boost - Dev mailing list wrote

...
The intention of this is to simplify managing a docker build environment for boost [...] This partially came out of the discussion we had a number of months back about how we handle third party dependencies. For the most part one has to look at README files in various locations to learn about them. This is an attempt to pull all the dependencies together for a complete build into one place (the Dockerfile).

James,

TBH, to me, this gives quite a little idea what is the actual purpose and target audience of the docker images.

Is it for Boost users, to provide them with Boost itself? Is it for Boost contributors, to provide them with development and testing environment? What do you mean by "all the dependencies", Boost itself or deps required by Boost?

Best regards, Mateusz Loskot

It allows for easier boost development and release engineering by providing docker images that are capable of building, testing, and packaging boost on a variety of platforms. Right now two flavors of Debian and Ubuntu each are provided, and I'd like to get some RedHat variants in there. Therefore it is really designed for folks working on boost in some way, not those consuming it for their apps, although that is possible and will make it easier for folks that must build from source to meet requirements for their projects. The dependencies are the third party libraries that different boost modules use - some optionally, and some required, but all are included. In addition the environment to build the documentation is set up as part of the docker image so one has a complete build environment, however I'm not aware of any distro- specific packaging provided by boost itself, so that exercise is left up to the distro consumers. It came about because I was setting up my 5th or 6th boost development virtual machine due to hardware changes and I decided I didn't feel like going through it again. Now I can just install docker, build the docker image for the distro I want to work in, and off I go. I wasn't aware of the teeks99 dockerfile repository at the time. Given clang and gcc allow for layered installation, it would be easy to modify the dockerfiles here to include all of those compilers in one image, and then folks could use "toolset=" to select the one they want - however the clang toolset selector logic in b2 is substandard compared to gcc, so that needs to be worked on (I filed an issue in Boost.Build for this). Ideally I'd like to be able to provide a windows based docker build environment to make it easier to build on windows with all the required and optional third party libraries needed there. It's a lot more work to set up a windows build environment, but until Microsoft figures out they need to provide Visual Studio Build Tools that work on Windows Nano (<=600MB) instead of Windows Core (>3 GB), it's not that compelling. Of course, a complete environment for windows includes all of the compilers (MSVC 10, 11, 12, 14, 14.*) ideally so that one can build against all of them to make sure things work. This is what we've been leveraging in Appveyor through our repositories with CI. Having this (docker images and helper scripts) available in the superproject seems appropriate. Certainly the docker files, less certain about the shell scripts, since people usually customize them anyway, however a fair amount of work went into discovering the right set of command-line options to docker that allow one to debug inside the container effectively. The workflow I use is to have one shell in the container and one outside the container. The shell outside is used for git commands where your git identity is stored ( , although we could add another docker virtual mount that puts your user-specific git config into the container. Interesting...) jking@ubuntu:~$ cd boost jking@ubuntu:~/boost$ docker/run.sh ubuntu bionic boost@286b1533dac3:/boost$ cd libs/uuid boost@286b1533dac3:/boost/libs/uuid$ ../../b2 -q By default in the Ubuntu Bionic environment this will use the default toolset gcc which is gcc-7.3.0 in that environment. If you use the "debian stretch" environment it has gcc-6.3.0 in it. These are the default compilers that come with that distro and it is what consumers of boost will likely use. - Jim

Mateusz Loskot

11:40 a.m.

On 25 July 2018 at 12:50, James E. King III via Boost wrote:

...

On Wed, Jul 25, 2018 at 5:41 AM Mateusz Loskot via Boost < boost@lists.boost.org> wrote:

...
Boost - Dev mailing list wrote

...
The intention of this is to simplify managing a docker build environment for boost [...] This partially came out of the discussion we had a number of months back about how we handle third party dependencies. For the most part one has to look at README files in various locations to learn about them. This is an attempt to pull all the dependencies together for a complete build into one place (the Dockerfile).

James,

TBH, to me, this gives quite a little idea what is the actual purpose and target audience of the docker images.

Is it for Boost users, to provide them with Boost itself? Is it for Boost contributors, to provide them with development and testing environment? What do you mean by "all the dependencies", Boost itself or deps required by Boost?

It allows for easier boost development and release engineering by providing docker images that are capable of building, testing, and packaging boost on a variety of platforms. Right now two flavors of Debian and Ubuntu each are provided, and I'd like to get some RedHat variants in there.

Therefore it is really designed for folks working on boost in some way

That is exactly what I'm looking for. The goal seems to be similar to Tom's.

...

The dependencies are the third party libraries that different boost modules use - some optionally, and some required, but all are included.

eg. libjpeg, libpng for GIL, zlib for others, etc. Makes sense.

...

It came about because I was setting up my 5th or 6th boost development virtual machine due to hardware changes and I decided I didn't feel like going through it again.

I've been using Vagrant for long time, but docker seems suitable for that purpose too, especially after I experienced it via multi-docker build workflow on CircleCI 2.0 (already used for Geometry and GIL).

...

I wasn't aware of the teeks99 dockerfile repository at the time. [...]

Ideally, if the community could agree to maintain single boostorg/docker I'd like to be able to pull docker image(s), run regression tests and submit to the Boost waterfall. I'd also like to get some *BSD images - especially after long standing failures of GIL on those systems. Ideally, if I could run a script overnight that pulls images one by one runs the tests and submits the results. I'm sure such automation is feasible.

...

It's a lot more work to set up a windows build environment, but until Microsoft figures out they need to provide Visual Studio Build Tools that work on Windows Nano (<=600MB) [...]

I've been using Vagrant to provide various Windows-based development environments, and I feel the same pain.

...

Having this (docker images and helper scripts) available in the superproject seems appropriate.

Yes.

...

Certainly the docker files, less certain about the shell scripts, since people usually customize them anyway, however a fair amount of work went into discovering the right set of command-line options to docker that allow one to debug inside the container effectively. [...]

If there is boosorg/docker, then there will be documentation on boost.org. I'll try to help. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Olaf van der Spek

10:14 a.m.

On Tue, May 8, 2018 at 1:59 PM, James E. King, III via Boost wrote:

...

One day when Microsoft supports Visual Studio compilation inside a reasonably sized windows container (i.e. based on nanoserver and not Windows Server Core), I'd like to add that as well since managing the dependencies on Windows is a bit harder (you usually have to build them).

./vcpkg install zlib isn't that hard is it? -- Olaf

2315

Age (days ago)

2394

Last active (days ago)

List overview

Download

8 comments

5 participants

participants (5)

James E. King III
James E. King, III
Mateusz Loskot
Olaf van der Spek
Tom Kent