[Serialization] Linker error on BOOST_CLASS_EXPORT upgrading from 1.44.0
Hello, I am facing a problem with serialization as implemented in Boost 1.47.0 which I did not experience in version 1.44.0. The problem appears as a series of linker errors (Visual C++ 8.0 (VS2005), WinXP 32bit): 1>Compiling... 1>Model_SerializationXml.cpp 1>Model_SerializationTxt.cpp 1>Compiling manifest to resources... 1>Linking... 1>Model_SerializationXml.obj : error LNK2005: "public: static struct boost::archive::detail::extra_detail::guid_initializer<class sm::CompLoad_Test> const & const boost::archive::detail::extra_detail::init_guid<class sm::CompLoad_Test>::g" (?g@?$init_guid@VCompLoad_Test@sm@@@extra_detail@detail@archive@boost @@2ABU?$guid_initializer@VCompLoad_Test@sm@@@2345@B) already defined in Model_SerializationTxt.obj [...many more similar errors] The scenario is as follows: I have a considerable bunch of classes which must be serialized by means of both text and XML archives. To this purpose, I have two compilation units, "Model_SerializationTxt.cpp" and "Model_SerializationXml.cpp", as follows: ///////////////////////////////////////////////// // Model_SerializationTxt.cpp [...] #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include <boost/archive/archive_exception.hpp> [...] #include "SerClassExport.h" [...] ///////////////////////////////////////////////// ///////////////////////////////////////////////// // Model_SerializationXml.cpp [...] #include <boost/archive/xml_iarchive.hpp> #include <boost/archive/xml_oarchive.hpp> #include <boost/archive/archive_exception.hpp> [...] #include "SerClassExport.h" [...] ///////////////////////////////////////////////// ///////////////////////////////////////////////// // SerClassExport.h [...] // APPLICATION_DEPENDENT: Include here files // relevant to model classes to be exported // for serialization #include "Model_Root.h" #include "SM/SM_Node.h" [...Many more class definition headers] [...] #include <boost/serialization/export.hpp> // APPLICATION_DEPENDENT: Include here files // relevant to model classes to be exported // for serialization BOOST_CLASS_EXPORT(model::Root) BOOST_CLASS_EXPORT(sm::Node) [...Many more class export definitions] [...] ///////////////////////////////////////////////// The problem seems to be related to the fact that BOOST_CLASS_EXPORT is defined twice in my project. In Boost 1.44.0, struct guid_initializer was defined in "export.hpp" inside an anonymous namespace, while in Boost 1.47.0 - actually, since version 1.45.0 - the definition is inside namespace extra_detail, even if the comment above is unchanged: ///////////////////////////////////////////////// // export.hpp in Boost 1.44.0 [...] // Note INTENTIONAL usage of anonymous namespace in header. // This was made this way so that export.hpp could be included // in other headers. This is still under study. namespace { template<class T> struct guid_initializer [...] #define BOOST_CLASS_EXPORT_IMPLEMENT(T)\ namespace boost {\ namespace archive {\ namespace detail {\ namespace {\ template<>\ struct init_guid< T > {\ static guid_initializer< T > const & g;\ };\ guid_initializer< T > const & init_guid< T >::g =\ ::boost::serialization::singleton<\ guid_initializer< T >\ >::get_mutable_instance().export_guid();\ }}}}\ [...] ///////////////////////////////////////////////// ///////////////////////////////////////////////// // export.hpp in Boost 1.47.0 [...] // Note INTENTIONAL usage of anonymous namespace in header. // This was made this way so that export.hpp could be included // in other headers. This is still under study. namespace extra_detail { template<class T> struct guid_initializer [...] #define BOOST_CLASS_EXPORT_IMPLEMENT(T)\ namespace boost {\ namespace archive {\ namespace detail {\ namespace extra_detail {\ template<>\ struct init_guid< T > {\ static guid_initializer< T > const & g;\ };\ guid_initializer< T > const & init_guid< T >::g =\ ::boost::serialization::singleton<\ guid_initializer< T >\ >::get_mutable_instance().export_guid();\ }}}}\ [...] ///////////////////////////////////////////////// I guess that this might have a role in originating the linker problem. How can I avoid this error? I already tried including the list of BOOST_CLASS_EXPORT only once, i.e in Model_SerializationTxt.cpp only. This solved the linker problem, but serialization of data in XML was then impossible (runtime error). Unfortunately, merging the two compilation units in one is not possible, because the compiler (Visual C++ 8.0) goes out of memory while compiling the big unit (process memory grows beyond 2GB). Is there, in your opinion, a solution or a workaround to this situation? Let me apologize for the very long mail, I did not succeed in shortening it further. Looking forward to any possible advice, thank you in advance for your attention. Enrico
Enrico Carrara wrote:
Looking forward to any possible advice, thank you in advance for your attention.
Look at the most recent documentation. BOOST_CLASS_EXPORT is now replaced with two different macros (BOOST_CLASS_EXPORT_DECLARE and DEFINE or something like that) This permits one to arrange his code so that multiple definitions are avoided. Robert Ramey
On Mon, Jul 25, 2011 at 6:30 PM, Robert Ramey <ramey@rrsd.com> wrote:
Look at the most recent documentation. BOOST_CLASS_EXPORT is now replaced with two different macros (BOOST_CLASS_EXPORT_DECLARE and DEFINE or something like that) This permits one to arrange his code so that multiple definitions are avoided.
Thank you very much for your answer. I tried using the two new macros (BOOST_CLASS_EXPORT_KEY and BOOST_CLASS_EXPORT_IMPLEMENT). I replaced all the previous BOOST_CLASS_EXPORT in SerClassExport.h with the corresponding _KEY, and this solved the linker problem. So far, so good. Then, in order to provided explicit code instantiation, I created a new translation unit, "SerClassExport.cpp", as follows: ///////////////////////////////////////////////// // SerClassExport.cpp #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include <boost/archive/xml_iarchive.hpp> #include <boost/archive/xml_oarchive.hpp> #include "Model_Root.h" #include "SM/SM_Node.h" [...Many other class definition headers includes] [...] #include <boost/serialization/export.hpp> BOOST_CLASS_EXPORT_IMPLEMENT(model::Root) BOOST_CLASS_EXPORT_IMPLEMENT(sm::Node) [...Many other class export definitions] [...] ///////////////////////////////////////////////// I did it this way because, as far as I understand, I must include all the archives before the _IMPLEMENT macros. Now, when compiling SerClassExport.cpp, the compiler again goes out of heap space, as it did before splitting the original unique .cpp into two translation units - one for each archive type. Am I doing it right? It seems that I did not catch how to have two separate translation units without redefinition of guid_initializer::g. Of course, if I remove the "extra_detail" namespace in export.hpp (coming back to an unnamed namespace as it was in 1.44.0), everything goes ok, but I can hardly think that this might be a feasible solution. Any idea on possible alternatives? Thank you in advance, Enrico
Enrico Carrara wrote:
On Mon, Jul 25, 2011 at 6:30 PM, Robert Ramey <ramey@rrsd.com> wrote:
Look at the most recent documentation. BOOST_CLASS_EXPORT is now replaced with two different macros (BOOST_CLASS_EXPORT_DECLARE and DEFINE or something like that) This permits one to arrange his code so that multiple definitions are avoided.
Thank you very much for your answer.
I tried using the two new macros (BOOST_CLASS_EXPORT_KEY and BOOST_CLASS_EXPORT_IMPLEMENT). I replaced all the previous BOOST_CLASS_EXPORT in SerClassExport.h with the corresponding _KEY, and this solved the linker problem. So far, so good.
Then, in order to provided explicit code instantiation, I created a new translation unit, "SerClassExport.cpp", as follows:
///////////////////////////////////////////////// // SerClassExport.cpp #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include <boost/archive/xml_iarchive.hpp> #include <boost/archive/xml_oarchive.hpp>
#include "Model_Root.h" #include "SM/SM_Node.h" [...Many other class definition headers includes] [...] #include <boost/serialization/export.hpp> BOOST_CLASS_EXPORT_IMPLEMENT(model::Root) BOOST_CLASS_EXPORT_IMPLEMENT(sm::Node) [...Many other class export definitions] [...] /////////////////////////////////////////////////
I did it this way because, as far as I understand, I must include all the archives before the _IMPLEMENT macros. Now, when compiling SerClassExport.cpp, the compiler again goes out of heap space, as it did before splitting the original unique .cpp into two translation units - one for each archive type.
Am I doing it right? It seems that I did not catch how to have two separate translation units without redefinition of guid_initializer::g.
Of course, if I remove the "extra_detail" namespace in export.hpp (coming back to an unnamed namespace as it was in 1.44.0), everything goes ok, but I can hardly think that this might be a feasible solution.
Any idea on possible alternatives? Thank you in advance, Enrico
I'm going to guess that there's too much instatiation going on. Try the following: a) try changing the headers so they don't inlcude inline functions. Put definitions in a separate cpp or ipp file. b) since a is a huge amount of work. Try this: build your above module a little at a time. That is - comment out most of the headers and try to build - of course it will fail to link. Add headers until it starts to have the problem. The basic goal is to find which templates are being "accdently instantiated multiple times. Robert Ramey
On Fri, Jul 29, 2011 at 6:35 PM, Robert Ramey <ramey@rrsd.com> wrote:
I'm going to guess that there's too much instatiation going on.
I agree
Try the following:
a) try changing the headers so they don't inlcude inline functions. Put
definitions in a separate cpp or ipp file.
Yes, I already have (almost) all inline code outside headers. I'll revise it further.
b) since a is a huge amount of work. Try this: build your above
module a little at a time. That is - comment out most of the headers and try to build - of course it will fail to link. Add headers until it starts to have the problem. The basic goal is to find which templates are being "accdently instantiated multiple times.
Doing as you wisely suggest, I noticed that the compiler's heap seems to blow up when trying to produce code for template member function "serialize" of some specific classes, which are are roots of containment structures made out of lists, vectors and maps of shared_ptr to other objects, which are in turn a composition of other objects. The containment structures is 3-4 levels deep. Think of it as a "Sketch" root object, containing a list of shared_ptr to "Polyline", each containing a list of shared_ptr to "LineSegment" or "ArcSegment", each containing a pair of "Point" and some more data. When the compiler parses/generates code for "Sketch::serialize", about two gigabytes of memory are allocated.
I found a workaround, anyway, tweaking some compiler optimizations. I also revised all the header files and moved some code outside, then isolated all BOOST_CLASS_EXPORT_IMPLEMENTs in one .cpp and moved any other serialization code in other two cpp. This seems to have solved my problem. Unfortunately, I haven't found a more effective strategy for reducing code bloat, so I am ready to accept/try any further advice in this regard. Thanks again for your advices. Enrico
Enrico Carrara wrote:
On Fri, Jul 29, 2011 at 6:35 PM, Robert Ramey <ramey@rrsd.com> wrote:
I found a workaround, anyway, tweaking some compiler optimizations. I also revised all the header files and moved some code outside, then isolated all BOOST_CLASS_EXPORT_IMPLEMENTs in one .cpp and moved any other serialization code in other two cpp. This seems to have solved my problem.
Unfortunately, I haven't found a more effective strategy for reducing code bloat, so I am ready to accept/try any further advice in this regard.
I don't think there is a more effective strategy for "code bloat". The problem is the flip side of inline code instantiation. On one hand you get faster code - (no call/return - deep optimization) on the other you get smaller code. This one of the main reasons that the serialization library is really two pieces - a pre-compiled library for all the code which doesn't depend on user types, and header library which does depend on user types. Generally this works pretty well. But if your data structures are deeply nested and serialization is used in multiple places - you're going to get code bloat. Happily there is a solution. As the application programmer you have the ability to decide which code you want to segregate into one module and which code you want to re-instantiate on demand. Downside is that you have to actively think about this - that's why we get paid the big bucks. Another issue crops up with libraries and especially DLLS. It's possible for instantiations to occur in differing modules. This might show up as multiple symbols in linking a static library and violations of ODR in DLLS. This latter can create problems with the serialization library which needs to keep a list of exported types. If the type is exported in more than one module, confusion can occur. I had a trap in the code to detect this latter - but it turns out to be too onerous a requirement that users avoid violating the One Definition Rule - so I had to disable the trip - for now. This turns out to be a surprisingly big subject which the C++ standard has yet to address - (I believe it's on the menu though). Robert Ramey
participants (1)
-
Enrico Carrara -
Robert Ramey