[Serialization] 1.33 RC1 nvp.hpp gotcha
I have found that the requirement that no serialization headers can be included before any archive headers means that you have to be careful which serialization headers you include in your class headers and indeed in what order you include them. In particular if you include serialization/nvp.hpp before serialization/vector.hpp and you use that class in another module that does not use archiving at all then that module will hit the error directive, example below. A solution would seem to be that serialization/nvp.hpp should only be included in the module that includes archive headers. I guess what I would like is some guidance on using serialization as it's become a bit more tricky in my experience. At least you can't willy-nilly include serialization headers in any order in any headers and expect the application to compile. Perhaps it only worked by luck before! For example, given class A below: class A { private: friend class boost::serialization::access; template <class ArchiveT> void serialize(ArchiveT &ar, unsigned int) { ar & BOOST_SERIALIZATION_NVP(m_Vector); } std::vector<int> m_Vector; }; What serialization headers should be included in A's header? What should be included in the header of the class that serializes A? My inclination would be to include serialization/nvp.hpp and serialization/vector.hpp since these are needed if (but only if) the template is instantiated. This is on the idea that a class should (generally) compile on it's own and you should not have to look to see what it is composed of to be able to get it to compile. Here's the example that does not work: B fails to compile with [C++ Fatal Error] basic_archive.hpp(21): F1003 Error directive: "no serialization headers my precede any archive headers" If you include nvp after vector, then it's ok too! A.h: ======= #include <vector> #include <boost/serialization/nvp.hpp> #include <boost/serialization/access.hpp> #include <boost/serialization/vector.hpp> class A { private: friend class boost::serialization::access; template <class ArchiveT> void serialize(ArchiveT &ar, unsigned int) { ar & BOOST_SERIALIZATION_NVP(m_Vector); } std::vector<int> m_Vector; }; ======= B.h: class B { public: B(); }; ======= B.cpp: #include "B.h" #include "A.h" B::B() { A InstanceOfA; } ======= Main.cpp #include <sstream> #include <boost/archive/text_oarchive.hpp> #include "A.h" #include "B.h" int main(int argc, char* argv[]) { A InstanceOfA; B InstanceOfB; std::stringstream ss; boost::archive::text_oarchive oa(ss); oa & InstanceOfA; return 0; }
Richard Jennings wrote:
I have found that the requirement that no serialization headers can be included before any archive headers means that you have to be careful which serialization headers you include in your class headers and indeed in what order you include them.
It is my intention that this issue be addressed with the simple rule: "all headers from the serialization directory should follow and headers from the archive directory" I believe that this rule should be very easy to follow. As a beneficial side-effect of flagging any usage of "archive" headers in include modules for classes. This would accidently link a serialization definition with implementation of a specific archive.
In particular if you include serialization/nvp.hpp before serialization/vector.hpp and you use that class in another module that does not use archiving at all then that module will hit the error directive, example below.
This would surprise me. Your example looks good to me. So I built on my own machine. Both main.cpp and B.cpp compiled without problems with VC 7.1 . So we're out of sync somehow.
A solution would seem to be that serialization/nvp.hpp should only be included in the module that includes archive headers.
This wouldn't work and shouldn't be necessary. Lets beat upon your example some more until we get the same results.
I guess what I would like is some guidance on using serialization as it's become a bit more tricky in my experience.
I would hope that its not too tricky. The system is designed to be as tricky as necessary - but no trickier.
At least you can't willy-nilly include serialization headers in any order in any headers and expect the application to compile.
True
Perhaps it only worked by luck before!
LOL - well I do consider myself a lucky person !!! But A very small observation on your code: #include <vector> // note this is superflous - but harmless #include <boost/serialization/vector.hpp> // as the above is included by this module Robert Ramey
Robert Ramey wrote:
Richard Jennings wrote:
I have found that the requirement that no serialization headers can be included before any archive headers means that you have to be careful which serialization headers you include in your class headers and indeed in what order you include them.
It is my intention that this issue be addressed with the simple rule:
"all headers from the serialization directory should follow and headers from the archive directory"
I believe that this rule should be very easy to follow.
You keep on making this statement, which is simply false. We've discussed this with you on the boost ml, and in the end result you've told that I should drop my design practice of including my headers first in implementation files, before any system and boost headers. That's basically a global change in style, and not so "easy to follow". - Volodya
Vladimir Prus wrote:
Robert Ramey wrote:
Richard Jennings wrote:
I have found that the requirement that no serialization headers can be included before any archive headers means that you have to be careful which serialization headers you include in your class headers and indeed in what order you include them.
It is my intention that this issue be addressed with the simple rule:
"all headers from the serialization directory should follow and headers from the archive directory"
I believe that this rule should be very easy to follow.
You keep on making this statement, which is simply false.
The statement "I believe that this rule should be very easy to follow" is definately true. I'm quite sure I know what I believe.
We've discussed this with you on the boost ml, and in the end result you've told that I should drop my design practice of including my headers first in implementation files, before any system and boost headers.
That's basically a global change in style, and not so "easy to follow".
The basic problem that has come up is the following: A.hpp ==== #include some serialization headers export, nvp, etc. class A { serialize(..) }; Before implementing this rule, any program which included A.hpp would require linking with the serialization library to build. This was provoked by the auto-linking facility. This would occur even if serialization wasn't used in the program. The made it amost impossible to have a "library" of classes which could be imported into various programs. This is a common and widely used style. which I have in mind when I write "I believe that this rule should be very easy to follow" . Given you're observations, I would probably rephrase that to "I believe that this rule should usually be very easy to follow" So. Now the question arises as to what can/should be done about this. a) I'm very much in agreement with the idea that the meer inclusion of a header shouldn't trigger the requirement of compilation against a library which isn't used. I realize that this is only a problem on compiles which support some sort of auto-link pragma. b) I'm don't think that imposition of the code organization as exemplified by A.hpp is a very heavy burden. Of course that's only my opinion. That is the rationale for the header ordering "rule". I can see now how this might conflict with some other "global style" but I'm not sure how to address it without creating another problem. That is, the imposition of this header ording rule was only to address a real problem for which I could find no better solution. I havn't seen your code and global style. But its hard for me to imagine a case where mixing archive headers and serialization headers in the same header file would be an attractive design choice. That's the real motivation for "I believe that this rule should usually be very easy to follow" Robert Ramey
Robert Ramey wrote:
The statement "I believe that this rule should be very easy to follow" is definately true. I'm quite sure I know what I believe.
Not when you start to employ pre-compiled headers. e.g. with 1.32 we were require to include serialization/shared_ptr before shared_ptr. We had shared_ptr in our pre-compiled header which meant we had to have serialization/shared_ptr in our pre-comiled header. But then this required us to put all archive types we may use above the serialization headers. This is undesirable but we have had to do it for 1.32 because of the shared_ptr problem. I understand the shared_ptr problem is gone with 1.33, but I'm just trying to point out that when you start using things like pre-compiled headers and such which are pretty much essential for build times of large scale projects rather than just unit tests, then header dependency orders can become a big issue and I believe these need addressing. A library that forces dependency orders on end users is not easy to use in large-scle projects IMHO and the issues we are hitting with serialization demonstrates that. Cheers Russell
Robert Ramey wrote:
Vladimir Prus wrote:
Robert Ramey wrote:
Richard Jennings wrote:
I have found that the requirement that no serialization headers can be included before any archive headers means that you have to be careful which serialization headers you include in your class headers and indeed in what order you include them.
It is my intention that this issue be addressed with the simple rule:
"all headers from the serialization directory should follow and headers from the archive directory"
I believe that this rule should be very easy to follow.
You keep on making this statement, which is simply false.
The statement "I believe that this rule should be very easy to follow" is definately true. I'm quite sure I know what I believe.
Oh ;-) I kinda though that "I believe" is just a polite way of making a statement.
We've discussed this with you on the boost ml, and in the end result you've told that I should drop my design practice of including my headers first in implementation files, before any system and boost headers.
That's basically a global change in style, and not so "easy to follow".
The basic problem that has come up is the following:
A.hpp ==== #include some serialization headers export, nvp, etc. class A { serialize(..) };
Before implementing this rule, any program which included A.hpp would require linking with the serialization library to build. This was provoked by the auto-linking facility. This would occur even if serialization wasn't used in the program. The made it amost impossible to have a "library" of classes which could be imported into various programs.
I don't get this. Why including <serialization/export.hpp> forces auto-linking? You can arrange the headers to that auto-linking is enabled only if user includes headers that require cpp files, no? Specifically, you should make sure that boost/serialization/config.hpp (that triggers auto-linking), is not included by export.hpp, either directly or indirectly.
a) I'm very much in agreement with the idea that the meer inclusion of a header shouldn't trigger the requirement of compilation against a library which isn't used. I realize that this is only a problem on compiles which support some sort of auto-link pragma.
I don't see how header order affects this. Can you elaborate? - Volodya
Vladimir Prus wrote:
The statement "I believe that this rule should be very easy to follow" is definately true. I'm quite sure I know what I believe.
Oh ;-) I kinda though that "I believe" is just a polite way of making a statement.
I mean to say that this is my opinion. I really do mean that, as far as I know, based on my personal experience, demos, and test cases I believe that organizing code subject to this rule/restriction shouldn't be difficult. I also mean that I'm open to be proved wrong via counter example. And the submission of such a counter example won't be taken personally in any way.
Before implementing this rule, any program which included A.hpp would require linking with the serialization library to build. This was provoked by the auto-linking facility. This would occur even if serialization wasn't used in the program. The made it amost impossible to have a "library" of classes which could be imported into various programs.
I don't get this. Why including <serialization/export.hpp> forces auto-linking? You can arrange the headers to that auto-linking is enabled only if user includes headers that require cpp files, no? Specifically, you should make sure that boost/serialization/config.hpp (that triggers auto-linking), is not included by export.hpp, either directly or indirectly.
...\more\separate_compilation.html describes the problem. Now if I recall, you're using gcc which doesn't support auto-linking so you would never see this problem. This addresses the seeming endless problems that users have getting their projects to link with the proper version of the library. The solution - auto-linking is that macros in the code invoke #pragmas which add data to the object files which indicate which libraries to include. These pragmas are invoked from the auto-linking header files regardless of whether or not there are unresolved references to one of the libraries. So just including a header file which uses auto-linking would trigger the requirement of a library being included in the link even though the library isn't actually used. I believe this issue would occur with any of the libraries which support auto-linking. I don't know if its an issue with other libraries. I would expect it to be an issue only with libraries which require that headers be included from within other headers.
a) I'm very much in agreement with the idea that the meer inclusion of a header shouldn't trigger the requirement of compilation against a library which isn't used. I realize that this is only a problem on compiles which support some sort of auto-link pragma.
I don't see how header order affects this. Can you elaborate?
In addressing this it occured to me that in the case where only a serialization header is included, it would be safe and desirable to skip over those parts of the headers which provoke auto-linking. This seemed very natural. It did imply that that any archive classes precede those headers. So it seemed simple and natural to generalize the "rule" already required by export.hpp. I really had in mind the usage scenario that was most common to me - that no header would include both archive and serialization headers. I didn't occur to me that this might cause a problem. The same "trick" is used by export to instantiate code for all archives and only those archives which have been previously included. The only alternatives I saw were: a) instantiate code for all archives - code bloat. b) require some sort of of defines or other user indicator as to which archives code should be instantiated for. This would be another hoop to jump through. So I opted for identifying the list of included archives as the list for which code should be instantiated for. The required that all archive classes precede the export.hpp. Now it has come about that this creates its own set of inconveniences for some applications. There's your case, and another has brought up precompiled headers. The auto-linking headers do contain a switch to turn off auto-linking. Perhaps even more attractive might be that this header inclusion rule be dependent on whether auto-linking is supported. On the gcc compiler - this would always be off. The concerns I would have about this are: a) it might mean an extra layer of complication in the serializaton headers which support auto-linking. I had to spend a significant amount of time to get this right and I don't relish making it more elaborate. b) It would mean that a program which used serialization on one platform (e.g. gcc) would not support auto-linking when moved to a platform which had auto-linking available. As I write this, this wouldn't be so bad - just one more section to the manual (the gotcha section) indicating that auto-linking should be suppressed via compile time define when the application is moved. Now someone has raised the issue of pre-compiled headers - which is a feature that as far as I know is only available on borland and Microsoft compilers which also support auto-linking. This is making my head hurt. So I'm not insensitive to your concerns. Its just that, like everything else, something simple ( just implement auto-link) a) turns out to be alot more complex than it starts out. Autolink for serialization required treating 3 separarte but inter-related modules. b) has non-obvious, non-trivial side-effects. c) is a lot more work that it would seem. Writing this makes the issues a lot clearer in my own mind and I will take another look at it. Robert Ramey
Robert Ramey wrote:
Now someone has raised the issue of pre-compiled headers - which is a feature that as far as I know is only available on borland and Microsoft compilers which also support auto-linking.
FYI: In my limited experience with it, Apple's GCC on OS X 10.3 and 10.4 supports pre-compiled headers also. Cheers Russell
Robert Ramey wrote:
Before implementing this rule, any program which included A.hpp would require linking with the serialization library to build. This was provoked by the auto-linking facility. This would occur even if serialization wasn't used in the program. The made it amost impossible to have a "library" of classes which could be imported into various programs.
I don't get this. Why including <serialization/export.hpp> forces auto-linking? You can arrange the headers to that auto-linking is enabled only if user includes headers that require cpp files, no? Specifically, you should make sure that boost/serialization/config.hpp (that triggers auto-linking), is not included by export.hpp, either directly or indirectly.
...\more\separate_compilation.html describes the problem.
Now if I recall, you're using gcc which doesn't support auto-linking so you would never see this problem. This addresses the seeming endless problems that users have getting their projects to link with the proper version of the library. <....>
Yes, I know both the problem and what's autolinking is about.
invoke #pragmas which add data to the object files which indicate which libraries to include. These pragmas are invoked from the auto-linking header files regardless of whether or not there are unresolved references to one of the libraries. So just including a header file which uses auto-linking would trigger the requirement of a library being included in the link even though the library isn't actually used.
Yes, sure, unless you clearly decide which headers require linking and which not, and include the internal header with #pragma only from the header of the first kind. Yes, it can be *extra* messy, but for serialization/base_object.hpp or for serialization/nvp.hpp this should not be very hard.
a) I'm very much in agreement with the idea that the meer inclusion of a header shouldn't trigger the requirement of compilation against a library which isn't used. I realize that this is only a problem on compiles which support some sort of auto-link pragma.
I don't see how header order affects this. Can you elaborate?
In addressing this it occured to me that in the case where only a serialization header is included, it would be safe and desirable to skip over those parts of the headers which provoke auto-linking. This seemed very natural. It did imply that that any archive classes precede those headers. So it seemed simple and natural to generalize the "rule" already required by export.hpp. I really had in mind the usage scenario that was most common to me - that no header would include both archive and serialization headers. I didn't occur to me that this might cause a problem.
The same "trick" is used by export to instantiate code for all archives and only those archives which have been previously included. The only alternatives I saw were: a) instantiate code for all archives - code bloat. b) require some sort of of defines or other user indicator as to which archives code should be instantiated for. This would be another hoop to jump through. So I opted for identifying the list of included archives as the list for which code should be instantiated for. The required that all archive classes precede the export.hpp.
Oh.. Okay, I think I'm starting to understand what's going on. 1. For BOOST_EXPORT used in headers, you want the list of archive classes 2. So, you want archive headers to be included before the header that contain BOOST_EXPORT. 3. You generalize the above requirement to "all archive headers must be included before all serialization headers". Is my understanding correct? If so, then I'd say this: 1. I don't have any magic solution for "archive headers must be included before BOOST_EXPORT" problem. One possibility would be to disallow BOOST_CLASS_EXPORT in headers -- after all exporting makes sense only for concrete classes (non-abstract), and so they have some code, and some .cpp module to invoke BOOST_CLASS_EXPORT there. Another would be to auto-register with polymorphic_archive as I've suggested previously. (Oops, I think I forgot to reply to your last post on that topic). 2. I'm not really happy with generalization. I still believe (although may be wrong), that including <serialization/nvp.hpp> and many other <serialization/> headers is possible without invoking auto-linking. Since not users want BOOST_CLASS_EXPORT at all, or want it in headers, this will reduce the number of "what's up with header" order questions you get on this list.
The auto-linking headers do contain a switch to turn off auto-linking. Perhaps even more attractive might be that this header inclusion rule be dependent on whether auto-linking is supported. On the gcc compiler - this would always be off. The concerns I would have about this are: a) it might mean an extra layer of complication in the serializaton headers
Sorry, but it seems that the BOOST_CLASS_EXPORT issue exists regardless of auto-linking, no?
This is making my head hurt.
So I'm not insensitive to your concerns. Its just that, like everything else, something simple ( just implement auto-link) a) turns out to be alot more complex than it starts out. Autolink for serialization required treating 3 separarte but inter-related modules. b) has non-obvious, non-trivial side-effects. c) is a lot more work that it would seem.
Writing this makes the issues a lot clearer in my own mind and I will take another look at it.
And seems I've finally understood your rationale. - Volodya
Vladimir Prus wrote:
Oh.. Okay, I think I'm starting to understand what's going on. 1. For BOOST_EXPORT used in headers, you want the list of archive classes 2. So, you want archive headers to be included before the header that contain BOOST_EXPORT. 3. You generalize the above requirement to "all archive headers must be included before all serialization headers".
Is my understanding correct?
Yes.
If so, then I'd say this: 1. I don't have any magic solution for "archive headers must be included before BOOST_EXPORT" problem. One possibility would be to disallow BOOST_CLASS_EXPORT in headers -- after all exporting makes sense only for concrete classes (non-abstract), and so they have some code, and some .cpp module to invoke BOOST_CLASS_EXPORT there. Another would be to auto-register with polymorphic_archive as I've suggested previously. (Oops, I think I forgot to reply to your last post on that topic).
It has been my view that BOOST_CLASS_EXPORT is really sort of class trait - assign GUID to the class and as such it should be part of the header so that the module can be included by different programs and they will all share the same GUID. (GUID= Globally, Unique IDentifier). I went through extra effort to permit this. However, it doesn't have to be in the header. I don't see any conflict if you have it in some other module.
2. I'm not really happy with generalization. I still believe (although may be wrong), that including <serialization/nvp.hpp> and many other <serialization/> headers is possible without invoking auto-linking. Since not users want BOOST_CLASS_EXPORT at all, or want it in headers, this will reduce the number of "what's up with header" order questions you get on this list.
Sorry, but it seems that the BOOST_CLASS_EXPORT issue exists regardless of auto-linking, no?
This is correct and somewhat confusing. They are unrelated issues - but I "fixed" them with the same solution - imposing a requirement on header sequence. This makes them seem more related than they are.
And seems I've finally understood your rationale.
I'll add an explanation of this topic to the "rationale" section of the manual. This discussion has made me think about this a little bit more. I haven't considered the implications in detail, but my current thought is: a) note that BOOST_CLASS_EXPORT does not have to be in a header module. I'm pretty sure this is true but it should be checked. I think this would address one of your concerns. b) The #error .... for archive/serialization header inclusion can probably be made conditional on whether auto-linking is disabled. There is a macro defined or not BOOST_NO_AUTO_LINKING(?) which will suppress auto-linking. In that case the auto-link problem can't occur so no requirement for header inclusion need be enforced. I didn't have to make a huge number of changes to make auto-link work the way I wanted - it just took some time to sort out the side-effects and find the right places to change. So I would expect I can spice up the header inclusion with conditionals that depend upon auto-linking. So in your case, you can get the original behavior by foregoing auto-linking - which isn't available with your platform of choice (gcc?) anyhow. Also your code would still be portable to other compilers with restriction that the library to link to would have to be specified explicitly. I think this would be a satisfactory situation all around. Robert Ramey
Robert Ramey wrote:
A very small observation on your code:
#include <vector> // note this is superflous - but harmless #include <boost/serialization/vector.hpp> // as the above is included by this module
But this is assuming we know what headers other headers include. Yes you can find out by examining those headers, but you can't tell just from the name so I don't believe it should be regarded as superfluous. Cheers Russell
participants (1)
-
Richard Jennings
-
Robert Ramey -
Russell Hind -
Vladimir Prus