On Fri, 3 Apr 2020 at 16:53, Stefan Seefeld wrote:

Hi Jonathan, On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:

tools/build/src/tools/python.jam says:

# On *nix, we do not want to link either Boost.Python or Python extensions # to libpython, because the Python interpreter itself provides all those # symbols. If we linked to libpython, we would get duplicate symbols. So # declare two targets -- one for building extensions and another for # embedding.

Why would that give duplicate symbols? Wouldn't ELF symbol interposition mean that only one is used? Why would it be a problem to link to libpython?

Do all *nix variants use ELF these days ?

All the ones in common usage, yes. And for my purposes, I only care about one specific OS (Fedora Linux) which is definitely ELF.

In Fedora's Boost RPM packages we *do* link libboost_python.so to the system libpython (and we don't get any problems).

What is your rationale for this, given that (according to the above reasoning) symbols will be resolved from the application that's loading the module ?

The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667 That links to https://svn.boost.org/trac10/ticket/2615 which suggests other people agree that linking to libpython is correct. Dave Abrahams seems to be the origin of the "linking will cause duplicate symbols" issue, which might only be the case when the python interpreter is statically linked to libpython.a. Since linux distros usually (maybe always?) dynamically link the python interpreter to libpythonX.Ym.so, that isn't an issue, and Boost's behaviour is wrong, and Fedora and RHEL are probably not the only distros having to kluge around it. But to do this, we have

these two local patches: https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-l...

and then to make it link to libpython3.7m.so (rather than libpython3.7.so) we have this patch to pass the "m" into the Boost build: https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-a...

I'd prefer not to need these local hacks. Would the upstream Boost.Python consider making changes so that libboost_python.so can be (optionally) linked to libpython?

Is there a cleaner way to do this than those patches?

I fully agree that whatever seems the right choice of build (including link) process for a given platform, we (boost) should incorporate that into our own infrastructure, so downstream package maintainers don't need to patch it. I'll leave it to the Boost.Build maintainers to answer the technical details of that process.

I should note that the linked patches worked fine up to Boost 1.69.0 but since 1.70.0 they break the build somehow, so that the mpi.so python extensions are no longer built. I am unable to figure out the problem, because I find jamfiles completely impenetrable. I'd love to try using cmake instead, but that doesn't seem to be supported for Boost.MPI. Boost 1.70.0 and 1.71.0 fail to build with: ...updating 24 targets... common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71.0 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71.0 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71.0 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71.0 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.a ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.a ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.a ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.a ...updated 24 targets... + mkdir -p /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost + touch /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/__init__.py + mv /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/ mv: cannot stat '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so': No such file or directory And 1.72.0 fails even earlier, with what looks like a conflict between building Boost.Serialization with "python" and "python-for-extensions" properties (if I'm reading python.jam right, which is unlikely): + echo ============================= build openmpi-x86_64 ================== + ./b2 -d+2 -q -j8 --with-mpi --with-graph_parallel --build-dir=openmpi-x86_64 variant=release threading=multi debug-symbols=on pch=off python=3.8 stage Performing configuration checks - default address-model : 64-bit - default architecture : x86 warning: non-free usage requirements <threading>multi ignored warning: in main-target mpi at /builddir/build/BUILD/boost_1_72_0/tools/build/src/user-config.jam:6 warning: non-free usage requirements <threading>multi ignored warning: in main-target boost_mpi at libs/mpi/build/Jamfile.v2:120 warning: non-free usage requirements <threading>multi ignored warning: in main-target boost_mpi_python at libs/mpi/build/Jamfile.v2:145 error: Name clash for '

libboost_serialization.so.1.72.0' error: error: Tried to build the target twice, with property sets having error: these incompatible properties: error: error: - none error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config error: error: Please make sure to have consistent requirements for these error: properties everywhere in your project, especially for install error: targets.

On Fri, 3 Apr 2020 at 18:13, Niall Douglas via Boost wrote:

On 03/04/2020 17:17, Jonathan Wakely via Boost wrote:

...

The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667

That links to https://svn.boost.org/trac10/ticket/2615 which suggests other people agree that linking to libpython is correct. Dave Abrahams seems to be the origin of the "linking will cause duplicate symbols" issue, which might only be the case when the python interpreter is statically linked to libpython.a. Since linux distros usually (maybe always?) dynamically link the python interpreter to libpythonX.Ym.so, that isn't an issue, and Boost's behaviour is wrong, and Fedora and RHEL are probably not the only distros having to kluge around it.

Dave got that originally from me.

The problem was this:

1. Python extension A is built against Boost.Python v1.X.

2. Python extension B is built against Boost.Python v1.Y.

3. What happens if end user imports Python extensions A and B into the same Python?

If Boost.Python insists on a Python link for Extension A, and a different Python link for Extension B, now what happens?

The generally accepted hack was to load Python extensions using dlopen(RTLD_LOCAL), which Python accepted as the least worst of all options back in the early 2000s, and incorporated that into CPython as the default semantic for loading extensions. Therefore there is no global symbol resolution at play here, instead Boost.Python ends up loading multiple copies of libpython.so, which is for obvious reasons very bad. So we don't do that.

I have no idea what has happened since then, and all the above information may be out of date. But that was the rationale ~20 years ago or so. It made a lot of sense at that time because of the fundamental brokenness that is ELF. Note that neither PE nor MachO have any of these problems.

Since ~2002, I patched GCC with -fvisibility in ~2005, and Dave committed a patchset for exporting visible symbols for a good chunk of Boost. So it may now be possible to completely change how Boost.Python does things, and meet your request.

Additionally, in a distro like Fedora everything is built with the same versions of Boost and Python anyway.

On Fri, 3 Apr 2020 at 18:07, Peter Dimov via Boost wrote:

Jonathan Wakely wrote:

...

Boost 1.70.0 and 1.71.0 fail to build with: ...updating 24 targets... ... ...updated 24 targets... ... + mv

/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so

...

/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/

...

mv: cannot stat

'/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':

...

No such file or directory

This is not failure to build; the error comes from the script, it can't find mpi.so.

Right, but mpi.so doesn't appear to get built. Maybe it did, and is just not where our build script expects it, I should check that.

...

And 1.72.0 fails even earlier, with what looks like a conflict between building Boost.Serialization with "python" and "python-for-extensions" properties (if I'm reading python.jam right, which is unlikely): ... error: Name clash for

'

libboost_serialization.so.1.72.0'

...

error: error: Tried to build the target twice, with property sets having error: these incompatible properties: error: error: - none error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config

I don't think this is a conflict between `python` and `python-for-extensions`. Rather, it is a conflict between libboost_serialization.so by itself (without rpath), and libboost_serialization.so built as a dependency of libboost_mpi.so (for which someone tries to set rpath.)

Ah, we also patch out the setting of rpaths, maybe we need to make that happen in an additional place now. Thanks for the clues what to look at next.

What does user-config.jam contain?

import os ; local RPM_OPT_FLAGS = [ os.environ RPM_OPT_FLAGS ] ; local RPM_LD_FLAGS = [ os.environ RPM_LD_FLAGS ] ; using gcc : : : <compileflags>$(RPM_OPT_FLAGS) <linkflags>$(RPM_LD_FLAGS) ; using mpi ; using python : 3.8 : /usr/bin/python3 : /usr/include/python3.8m : : : ;

(Also... before spending too much time on this, consider trying 1.73 as well.)

I will, although I would be surprised if these problems have just gone away by themselves, given that they seem to be related to downstream customizations for Fedora, and so not reported upstream to Boost.

Jonathan Wakely wrote:

I should add that we build in stages, because we want to build Boost.MPI twice, for openmpi and MPICH, so we do a build without MPI first:

./b2 -d+2 -q \ --without-mpi --without-graph_parallel --build-dir=serial \ variant=release threading=multi debug-symbols=on pch=off \ python=3.8 \ stage

and then for each MPI backend we do:

./b2 -d+2 \ --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \ variant=release threading=multi debug-symbols=on pch=off \ python=3.8 stage

so that's probably why libboost_serialization ends up with different sets of properties.

That's why you weren't getting the error before 1.72, but not the reason why libboost_serialization ends up with the different property sets.

Jonathan Wakely wrote:

...

(Also... before spending too much time on this, consider trying 1.73 as well.)

I will, although I would be surprised if these problems have just gone away by themselves, given that they seem to be related to downstream customizations for Fedora, and so not reported upstream to Boost.

They won't... but new ones may appear :-) as we changed the directory for mpi.so because it wasn't appropriate for Boost to install a file named `mpi.so` into the default destination /usr/lib.

On Fri, 3 Apr 2020 at 17:29, Dimitri John Ledkov wrote:

Strong objection from me.

No python extensions nor applications that embed a python interpreter should link with libpythonX.Y. The modules are effectively plugins, and are compiled to be dlopened by the interpret and interpreter is the one that must provide the symbols.

How an interpreter has been loaded, can be different from how an application/extenion is linked, and thus symbols can be different. Duplicate symbols are not a problem from ELF when they are the same, it can become a problem when they are conflicting yet compatible. And at that point modules can fail to load. The current schema allows sharing compiled extensions across different builds of the python interpret that may be shipped by different vendors.

Neither Ubuntu nor Debian links libpythonX.Y for any extension modules, or apps. Please continue not linking against libpythonX.Y and keep the binaries with unresolved python symbols.

For example, _ssl.so compiled extensions shipped by the python itself has all the Py* symbols undefined, and is not linked against libpythonX.Y.

Boost upstream, should follow Python upstream and do what they recommend, which is not linking with libpythonX.Y.

That's fine as the default, why do you object to making it configurable, so that distros that do link python modules to libpython can do so for Boost.Python too? On Fedora, I see $ ldd /usr/lib64/python3.7/lib-dynload/_ssl.cpython-37m-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffde6583000) libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007fb5deefb000) libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007fb5dec1b000) * libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (0x00007fb5de8cf000)* libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb5de8ad000) libc.so.6 => /lib64/libc.so.6 (0x00007fb5de6e4000) libz.so.1 => /lib64/libz.so.1 (0x00007fb5de6ca000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fb5de6c1000) libutil.so.1 => /lib64/libutil.so.1 (0x00007fb5de6bc000) libm.so.6 => /lib64/libm.so.6 (0x00007fb5de576000) /lib64/ld-linux-x86-64.so.2 (0x00007fb5defed000)