Serialization and source code organization
Hi, I'm currently trying to use the new Boost 1.32 serialization library to save the state of a medium complex program (about 10.000 lines of code). The problem is that the serialization lib seems to require very close source code coupling: The translation unit that contains the load/save functions for the application has to see all serialize() method definitions (as they are templates). That means that I cannot rely on forward declarations, so I always have to include the respective header file. This slows down the compilation noticably and results in errors such as "C1204: compiler limit : internal structure overflow" and "C1055: compiler limit : out of keys" (VC++ 7.1), as more or less all significant headers are included in this translation unit. The documentation is aware of this kind of problem and recommends manual instantiations of the serialize() methods (in the PIMPL section). Do I get something wrong or is this the only solution? I don't really like manual template instantiations, but in this case I can't imagine another way to deal with it, except waiting for the next VC++ release and hoping for the best. Malte
"Malte Clasen" <news@copro.org> wrote in message news:co3bqn$2ii$1@sea.gmane.org...
Hi,
file. This slows down the compilation noticably and results in errors such as "C1204: compiler limit : internal structure overflow" and "C1055: compiler limit : out of keys" (VC++ 7.1), as more or less all significant headers are included in this translation unit.
manual template instantiations, but in this case I can't imagine another way to deal with it, except waiting for the next VC++ release and hoping for the best.
FWIW, I have a new library in the works which causes a lot of internal errors and compiler limit errors on VC7.1. I've found that VC8.0 handles the library much better than VC7.1, so there is hope. In the mean time, I've found that I can often prevent these errors by using the "Line Numbers Only" debugging option (/Zd), or by surrounding some of the code with #pragma component(mintypeinfo, on) ... #pragma component(mintypeinfo, off) Jonathan
Jonathan Turkanis wrote:
FWIW, I have a new library in the works which causes a lot of internal errors and compiler limit errors on VC7.1. I've found that VC8.0 handles the library much better than VC7.1, so there is hope.
That's good news :)
In the mean time, I've found that I can often prevent these errors by using the "Line Numbers Only" debugging option (/Zd), or by surrounding some of the code with
#pragma component(mintypeinfo, on) ... #pragma component(mintypeinfo, off)
This didn't help unfortunately. But the Pimpl style seems to be an usable alternative. Malte
The pimpl example demonstrates pretty much all I know about the issue. Advantage: saves lots and lots of compile time. Disadvantage: requires explicit instantiation - a slight pain in the neck. Possible idea. I've seen this used to good effect with other projects - one upto 500K lines long!!! make another project - a static library for ALL the serialization instantiations. Link against the Other project library. Only the code actually used will be included in the final exe. Actually I used this method when I have to work on something moderatly large even when serialization isn't being used. It helps keep things organized so I can understand it better. Robert Ramey "Malte Clasen" <news@copro.org> wrote in message news:co3bqn$2ii$1@sea.gmane.org...
Hi,
I'm currently trying to use the new Boost 1.32 serialization library to save the state of a medium complex program (about 10.000 lines of code). The problem is that the serialization lib seems to require very close source code coupling: The translation unit that contains the load/save functions for the application has to see all serialize() method definitions (as they are templates). That means that I cannot rely on forward declarations, so I always have to include the respective header file. This slows down the compilation noticably and results in errors such as "C1204: compiler limit : internal structure overflow" and "C1055: compiler limit : out of keys" (VC++ 7.1), as more or less all significant headers are included in this translation unit.
The documentation is aware of this kind of problem and recommends manual instantiations of the serialize() methods (in the PIMPL section). Do I get something wrong or is this the only solution? I don't really like manual template instantiations, but in this case I can't imagine another way to deal with it, except waiting for the next VC++ release and hoping for the best.
Malte
Robert Ramey wrote:
The pimpl example demonstrates pretty much all I know about the issue.
Ok, I'm using that one now and it compiles fine. But another issue appeared: The automatic serialization of nested std::vectors fails under certain circumstances. I managed to isolate the problem to the following case: ---------- // disable warnings that don't matter in this example #pragma warning(disable:4511 4512 4244 4100 4267) #include <boost/serialization/serialization.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/serialization/vector.hpp> #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include <fstream> #include <vector> class A { private: friend class boost::serialization::access; template<class Archive> void serialize(Archive &, const unsigned int) {} }; template<typename T> class B { private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int /*version*/) { //ar.register_type< std::vector<T> >(); // workaround A //ar & vT; // workaround B, part 1/2 ar & vvT; // this one fails } //std::vector<T> vT; // workaround B, part 2/2 std::vector< std::vector<T> > vvT; }; void save() { std::ofstream ofs("test.dat"); boost::archive::text_oarchive oa(ofs); B< boost::shared_ptr<A> > b_ptr_a; oa << b_ptr_a; } void load() { std::ifstream ifs("test.dat"); boost::archive::text_iarchive ia(ifs); B< boost::shared_ptr<A> > b_ptr_a; ia >> b_ptr_a; } ---------- The serialization of std::vector< std::vector<T> > fails to compile (VC++ 7.1) unless one of the workarounds is used, that means some kind of registration of std::vector<T>. Is that a known limitation? I found nothing in the documentation about this case, but perhaps I overlooked something. Malte
insert the following two lines at the beginning of your program: "Malte Clasen" <news@copro.org> wrote in message news:co5ubm$cbc$1@sea.gmane.org...
Robert Ramey wrote:
The pimpl example demonstrates pretty much all I know about the issue.
Ok, I'm using that one now and it compiles fine.
But another issue appeared: The automatic serialization of nested std::vectors fails under certain circumstances. I managed to isolate the problem to the following case:
----------
// disable warnings that don't matter in this example #pragma warning(disable:4511 4512 4244 4100 4267)
#include <boost/serialization/serialization.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/serialization/vector.hpp> #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp>
#include <fstream> #include <vector>
class A { private: friend class boost::serialization::access; template<class Archive> void serialize(Archive &, const unsigned int) {} };
template<typename T> class B { private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int /*version*/) { //ar.register_type< std::vector<T> >(); // workaround A //ar & vT; // workaround B, part 1/2 ar & vvT; // this one fails }
//std::vector<T> vT; // workaround B, part 2/2 std::vector< std::vector<T> > vvT; };
void save() { std::ofstream ofs("test.dat"); boost::archive::text_oarchive oa(ofs);
B< boost::shared_ptr<A> > b_ptr_a; oa << b_ptr_a; }
void load() { std::ifstream ifs("test.dat"); boost::archive::text_iarchive ia(ifs);
B< boost::shared_ptr<A> > b_ptr_a; ia >> b_ptr_a; }
----------
The serialization of std::vector< std::vector<T> > fails to compile (VC++ 7.1) unless one of the workarounds is used, that means some kind of registration of std::vector<T>. Is that a known limitation? I found nothing in the documentation about this case, but perhaps I overlooked something.
Malte
insert the following two lines at the beginng of your program #include <boost/config.hpp> #define BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP and your program will compile. The above two lines supresses usage of ADL with vc 7.1 . I'm not sure what the circumstances are so I can't make a small test case. I'm am coming to believe that ADL on VC 7.1 is broken. The solution above isn't a good one but it does illustrate the source of the problem. Good Luck, Robert Ramey "Malte Clasen" <news@copro.org> wrote in message news:co5ubm$cbc$1@sea.gmane.org...
Robert Ramey wrote:
The pimpl example demonstrates pretty much all I know about the issue.
Ok, I'm using that one now and it compiles fine.
But another issue appeared: The automatic serialization of nested std::vectors fails under certain circumstances. I managed to isolate the problem to the following case:
----------
// disable warnings that don't matter in this example #pragma warning(disable:4511 4512 4244 4100 4267)
#include <boost/serialization/serialization.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/serialization/vector.hpp> #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp>
#include <fstream> #include <vector>
class A { private: friend class boost::serialization::access; template<class Archive> void serialize(Archive &, const unsigned int) {} };
template<typename T> class B { private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int /*version*/) { //ar.register_type< std::vector<T> >(); // workaround A //ar & vT; // workaround B, part 1/2 ar & vvT; // this one fails }
//std::vector<T> vT; // workaround B, part 2/2 std::vector< std::vector<T> > vvT; };
void save() { std::ofstream ofs("test.dat"); boost::archive::text_oarchive oa(ofs);
B< boost::shared_ptr<A> > b_ptr_a; oa << b_ptr_a; }
void load() { std::ifstream ifs("test.dat"); boost::archive::text_iarchive ia(ifs);
B< boost::shared_ptr<A> > b_ptr_a; ia >> b_ptr_a; }
----------
The serialization of std::vector< std::vector<T> > fails to compile (VC++ 7.1) unless one of the workarounds is used, that means some kind of registration of std::vector<T>. Is that a known limitation? I found nothing in the documentation about this case, but perhaps I overlooked something.
Malte
Just to confirm that I encountered the same problem with serialization/vc7.1. My workaround (after 2 day's of playing around) was to actually *use* the serialization template in every classes. That is to say, other than template<class Archive> void serialize(Archive & ar, const unsigned int /*version*/) ... I define some "saveThisClass" function to oa<<*this, although this function will never be used. Please consider adding #define BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP to boost::serialization document. Bo On Thu, 25 Nov 2004 21:44:40 -0800, Robert Ramey <ramey@rrsd.com> wrote:
insert the following two lines at the beginng of your program
#include <boost/config.hpp> #define BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP
and your program will compile.
The above two lines supresses usage of ADL with vc 7.1 . I'm not sure what the circumstances are so I can't make a small test case. I'm am coming to believe that ADL on VC 7.1 is broken. The solution above isn't a good one but it does illustrate the source of the problem.
Good Luck,
Actually, I'm making alterations so that the library functioning will better work with and without compilers that implement any combination of ADL and two-phase lookup. I hope this solves the issue once and for all. Robert Ramey "Bo Peng" <ben.bob@gmail.com> wrote in message news:6ea7b543041128153362decf8b@mail.gmail.com...
Just to confirm that I encountered the same problem with serialization/vc7.1. My workaround (after 2 day's of playing around) was to actually *use* the serialization template in every classes. That is to say, other than template<class Archive> void serialize(Archive & ar, const unsigned int /*version*/) ... I define some "saveThisClass" function to oa<<*this, although this function will never be used.
Please consider adding #define BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP to boost::serialization document.
Bo
On Thu, 25 Nov 2004 21:44:40 -0800, Robert Ramey <ramey@rrsd.com> wrote:
insert the following two lines at the beginng of your program
#include <boost/config.hpp> #define BOOST_NO_ARGUMENT_DEPENDENT_LOOKUP
and your program will compile.
The above two lines supresses usage of ADL with vc 7.1 . I'm not sure what the circumstances are so I can't make a small test case. I'm am coming to believe that ADL on VC 7.1 is broken. The solution above isn't a good one but it does illustrate the source of the problem.
Good Luck,
participants (1)
-
Bo Peng -
Jonathan Turkanis -
Malte Clasen -
Robert Ramey