On Fri, 3 Apr 2020 at 16:53, Stefan Seefeld
Hi Jonathan, On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:
tools/build/src/tools/python.jam says:
# On *nix, we do not want to link either Boost.Python or Python extensions # to libpython, because the Python interpreter itself provides all those # symbols. If we linked to libpython, we would get duplicate symbols. So # declare two targets -- one for building extensions and another for # embedding.
Why would that give duplicate symbols? Wouldn't ELF symbol interposition mean that only one is used? Why would it be a problem to link to libpython?
Do all *nix variants use ELF these days ?
All the ones in common usage, yes. And for my purposes, I only care about one specific OS (Fedora Linux) which is definitely ELF.
In Fedora's Boost RPM packages we *do* link libboost_python.so to the system libpython (and we don't get any problems).
What is your rationale for this, given that (according to the above reasoning) symbols will be resolved from the application that's loading the module ?
The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667 That links to https://svn.boost.org/trac10/ticket/2615 which suggests other people agree that linking to libpython is correct. Dave Abrahams seems to be the origin of the "linking will cause duplicate symbols" issue, which might only be the case when the python interpreter is statically linked to libpython.a. Since linux distros usually (maybe always?) dynamically link the python interpreter to libpythonX.Ym.so, that isn't an issue, and Boost's behaviour is wrong, and Fedora and RHEL are probably not the only distros having to kluge around it. But to do this, we have
these two local patches: https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-l...
and then to make it link to libpython3.7m.so (rather than libpython3.7.so) we have this patch to pass the "m" into the Boost build: https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-a...
I'd prefer not to need these local hacks. Would the upstream Boost.Python consider making changes so that libboost_python.so can be (optionally) linked to libpython?
Is there a cleaner way to do this than those patches?
I fully agree that whatever seems the right choice of build (including link) process for a given platform, we (boost) should incorporate that into our own infrastructure, so downstream package maintainers don't need to patch it. I'll leave it to the Boost.Build maintainers to answer the technical details of that process.
I should note that the linked patches worked fine up to Boost 1.69.0 but since 1.70.0 they break the build somehow, so that the mpi.so python extensions are no longer built. I am unable to figure out the problem, because I find jamfiles completely impenetrable. I'd love to try using cmake instead, but that doesn't seem to be supported for Boost.MPI. Boost 1.70.0 and 1.71.0 fail to build with: ...updating 24 targets... common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71.0 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71.0 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71.0 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71.0 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.a ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.a ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.a ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1 ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71 common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.a ...updated 24 targets... + mkdir -p /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost + touch /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/__init__.py + mv /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/ mv: cannot stat '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so': No such file or directory And 1.72.0 fails even earlier, with what looks like a conflict between building Boost.Serialization with "python" and "python-for-extensions" properties (if I'm reading python.jam right, which is unlikely): + echo ============================= build openmpi-x86_64 ================== + ./b2 -d+2 -q -j8 --with-mpi --with-graph_parallel --build-dir=openmpi-x86_64 variant=release threading=multi debug-symbols=on pch=off python=3.8 stage Performing configuration checks - default address-model : 64-bit - default architecture : x86 warning: non-free usage requirements <threading>multi ignored warning: in main-target mpi at /builddir/build/BUILD/boost_1_72_0/tools/build/src/user-config.jam:6 warning: non-free usage requirements <threading>multi ignored warning: in main-target boost_mpi at libs/mpi/build/Jamfile.v2:120 warning: non-free usage requirements <threading>multi ignored warning: in main-target boost_mpi_python at libs/mpi/build/Jamfile.v2:145 error: Name clash for '
libboost_serialization.so.1.72.0' error: error: Tried to build the target twice, with property sets having error: these incompatible properties: error: error: - none error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config error: error: Please make sure to have consistent requirements for these error: properties everywhere in your project, especially for install error: targets.