[serialization] proposed improvements: forward-compatibility of serialization
Hello Everybody, (it seems that my previous post had contained only title - so I repeat the full posting below) We've started (again) using boost::serialization in our multi-platform distributed application about one year ago. However, we have arrived at a road block with forward-compatibility of serialization - which of major importance to us since we cannot force users to upgrade to the same/newer version of application all at the same time. Firstly, question is: perhaps you or someone that you know of - would be interested in such improvement work? We are small company rushed against some die-or-prosper deadlines - and simply do not have enough resources to do it on our own in time avialable - but we could sponsor such work. Secondly, the problem is: To avoid ambiguity I have put together the following description and our own suggestion for solution: The project requirement is centred on improving multi-version class compatibility in binary archives e.g. more flexibility in a situation when we are reading or transferring modified class in C++ code. We need to go beyond what current version of object versioning in boost offers: http://www.boost.org/doc/libs/1_55_0/libs/serialization/doc/tutorial.html#ve... Specifically we need to add ability to skip reading unknown or unexpected fields in a non-XML archives we are parsing. Specific example: Version of the application (newer one) that saved the archive has the following code: struct extensionClass { std::string moreInfo; time_t date; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & moreInfo; ar & date; } } class gps_position { public: template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & degrees; ar & minutes; ar & seconds; ar & new_field } int degrees; int minutes; float seconds; extensionClass new_field; gps_position(){}; }; And now the version of the software that needs to read the archive is an older application that has the code that implements the gps_position class in the following way: class gps_position { public: template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & degrees; ar & minutes; ar & seconds; } int degrees; int minutes; float seconds; gps_position(){}; }; The purpose of the project is to gracefully allow for serialization in older version of application to continue reading the archive, by simply omitting information related to extensionClass new_field; e.g. to properly advance the read position to the beginning of next class in the archive. It is desirable to add to programming interface information that there was incompatibility detected, for further escalation to either to user interface, or higher level code logic that might decide, for instance, to change communication protocol. It is critically important for us to optimize archive size while adding this capability. We prefer to use eos::portable_iarchive which provides varinteger support for multi-OS compatibility (size of int, little vs big endian), therefore we prefer solution which would accept any archive, however, we will be ok with standard boost binary archive solution. Thirdly, the proposal is: The most promising solution that we envisage is to add recursive size information associated with the number identifying class (already placed in front of every data object). By recursive size information I mean size not in terms of bytes for raw data, but in terms of number of fields/objects in the class. The size information would than be expanded until it can be expressed in terms of C++11 standard predefined POD sizes. For instance for simple class example: class SimplePODs { uint32_t firstField, uint64_t int secondField; float thirdField; } The associated (leading) size information for the SimplePODs class is 3. Than within the class itself we already have identifiers of each field types. Since they are POD we can maintain global (and for the archive) dictionary of sizes for each identifier in this case: 4 than 8 and 4 And than it follows: class AggregateClass { SimplePODs first_field; char second_field; } In this case the size information (in the global dictionary) for identifier associated with AggregateClass is 2. Than within the class itself sizes for first_field would be 3 and for second_field it is of course 1. I believe this can be done as part of the version and object tracking process so the performance would still be high and most importantly, the incremental size overhead both in memory and in the archive will be relatively small (only one size information per each type of object contained in the archive - and no size overhead for POD fields). By the way this approach would also increase multi-OS compatibility for instance class on the source computer might be 32-bit compilation of : class CrossPlatform { int idont_care_about_size; } And during saving that field: idont_care_about_size would actually be saved and identified as uint32_t The same CrossPlatform class on destination machine would be compiled as 64-bit application - but since target int would be larger that source we can do silent promotion. Of course the other way around, we can throw exception if the actual value contained within idont_care_about_size exceeds the targets' compilation POD size. OK - I hope that my explanation is clear and I would appreciate any feedback that you might have. And finally let me state that we do greatly appreciate (and use) a lot of boost work and in particular, we consider boost.serialization approach to be probably the best possible under the current constrains of the language. Best regards, Andrew Horoszczak
Andrzej Horoszczak
Hello Everybody,
(it seems that my previous post had contained only title - so I repeat the full posting below)
We've started (again) using boost::serialization in our multi-platform distributed application about one year ago. However, we have arrived at a road block with forward-compatibility of serialization - which of major importance to us since we cannot force users to upgrade to the same/newer version of application all at the same time.
If you are not already aware of it, you might want to look into Google Protocol Buffers. It doesn't directly interface with existing C++ types as conveniently as Boost Serialization, but it is specifically designed to support forward and backward compatibility, and I believe provides all of the functionality you are looking for.
On Tue, Apr 15, 2014 at 5:14 AM, Jeremy Maitin-Shepard
If you are not already aware of it, you might want to look into Google Protocol Buffers. It doesn't directly interface with existing C++ types as conveniently as Boost Serialization, but it is specifically designed to support forward and backward compatibility, and I believe provides all of the functionality you are looking for.
Also http://kentonv.github.io/capnproto/ (from the same original dev of CapnProto) however it don't support visual studio yet.
On 04/15/2014 05:14 AM, Jeremy Maitin-Shepard wrote:
If you are not already aware of it, you might want to look into Google Protocol Buffers. It doesn't directly interface with existing C++ types as conveniently as Boost Serialization, but it is specifically designed to support forward and backward compatibility, and I believe provides all of the functionality you are looking for.
I was also thinking Protocol Buffers when I read the original post. Unlike the size-based proposal in the original post, Protocol Buffers also handles the case where you remove parameters from the API. While you are correct that Protocol Buffers has a different interface than Boost.Serialization, I believe that it is possible to create a Boost.Serialization archive for Protocol Buffers, although we would have to introduce an index-value pair, similar to the named-value pair.
Andrzej Horoszczak писал 2014-04-14 13:15: Hi, YAS(https://github.com/niXman/yas) ? -- Regards, niXman ___________________________________________________ Dual-target(32 & 64-bit) MinGW-W64 compilers for 32 and 64-bit Windows: http://sourceforge.net/projects/mingw-w64/ ___________________________________________________ Another online IDE: http://liveworkspace.org/
On 2014-04-14, 5:15 AM, Andrzej Horoszczak wrote:
Hello Everybody,
(it seems that my previous post had contained only title - so I repeat the full posting below)
We've started (again) using boost::serialization in our multi-platform distributed application about one year ago. However, we have arrived at a road block with forward-compatibility of serialization - which of major importance to us since we cannot force users to upgrade to the same/newer version of application all at the same time.
This was a problem that interested me a few years ago so I made a solution that worked with any archive and I presented it at BoostCon (now CppNow). It was inefficient, hacky and required no modifications to existing archives but forward compatibility did work so I know it is completely possible. The proposal itself doesn't seem impossible, at a glance. My immediate question would be whether you already have released code for the products in question which complicates things a bit. Sohail
participants (6)
-
Andrzej Horoszczak
-
Bjorn Reese
-
Jeremy Maitin-Shepard
-
Klaim - Joël Lamotte
-
niXman
-
Sohail Somani