Re: [Boost-users] Serialization newbie needs help
"Robert Ramey" <ramey@rrsd.com> wrote:
Actually, I've considered reviewing all the email in this list and summarizing the results in a section "Hints and Troubleshooting Tips". I think this is useful and necessary and useful, but I just havent gotten around to it.
That sounds ideal.
But if you could provide me an example of when pointer serialization shines, then I would say "Aha! Now I understand why Robert went to so much trouble to make this work.", and I could be at peace with the world again! :-)
LOL - I haven't had time to make any real apps lately.
So... no example at all? My problem is that I just don't see when pointer serialization would be a big win. But I can't tell if I just have a limited perspective. That's why I was genuinely hoping that you could provide a concrete example. Naturally, a system that "just works" no matter what structure is thrown at it sounds very enticing. However, as you have acknowledged, the system is not foolproof, and user gotchas remain. One way to eliminate the gotchas would be to reduce the scope of the library. Clearly, this is unlikely to happen, as (i) you've invested a lot of time and effort getting the pointer magic to work, and (ii) there are probably a lot of applications out there that would break if you did so. I for one would certainly appreciate a simplified version which omitted the pointer magic, and thus removed the const trap and similar issues. If I want to set up pointers within an object, then I can do so manually using the type-specific load()/save() functions. Obviously, I can just switch to a different serialization library (e.g. s11n). But as much as possible, I would like to stick with whatever is most "standard". And as part of the Boost family, I'm guessing that boost::serialization is top dog here. Would that be right? Is this library included in any of the proposed C++ standards (TR2, C++0x)?
BTW, I don't want to sound like I'm bashing your library, so let me end with a compliment. I think the symmetric design of serialize(), with the same function used for both input and output, is one of the prettiest idioms I've seen in years.
And that's the one thing I didn't do - I believe I "borrowed" it from Jens Maurer.
Hah! Ooops. Well, that compliment backfired! ;-) Let's try again: I *really* appreciate the built-in support for all the STL container classes. So far, these have "just worked" perfectly for me. Nice work!
One tiny point. I'm aware that the "const" trap is annoying. But I've become convinced that the annoyance is small compared to the hassles it saves me and users from. I suspect that those who have a problem with aren't in the habit of using "const" everywhere they can.
It does seem bizarre to have to const-cast an object just so that it can be output. I presume that the majority of objects that are to be serialized are not intrinsically constant, so this needs to be done the majority of the time. And it looks dangerous. What does the const-cast actually mean in this context? I assume it means: "I promise that no other thread is going to modify the object (or anything its pointers point to) while the serialize() operation is pending. Is that correct? If so, then shouldn't it be solved explicitly with concurrent programming primitives (i.e. mutexes or similar)? -- Dominick
Dominick Layfield wrote:
"Robert Ramey" <ramey@rrsd.com> wrote:
Actually, I've considered reviewing all the email in this list and summarizing the results in a section "Hints and Troubleshooting Tips". I think this is useful and necessary and useful, but I just havent gotten around to it.
That sounds ideal.
But if you could provide me an example of when pointer serialization shines, then I would say "Aha! Now I understand why Robert went to so much trouble to make this work.", and I could be at peace with the world again! :-)
LOL - I haven't had time to make any real apps lately.
So... no example at all?
My problem is that I just don't see when pointer serialization would be a big win. But I can't tell if I just have a limited perspective. That's why I was genuinely hoping that you could provide a concrete example.
There are a couple of demos in the "example" directory which demonstrate this facility. The are small examples for illustrative purposes, but they are not contrived and represent the kinds of things that are interesting for people to do.
Naturally, a system that "just works" no matter what structure is thrown at it sounds very enticing.
This is the motivation from my personal perspective. Also, any library submission to boost must compare favorably with in every aspect with any alternative system known to man. Otherwise someone in the review process will make a huge deal that this one feature is so important that lack there of should disqualify the library for acceptance.
However, as you have acknowledged, the system is not foolproof, and user gotchas remain.
No system is foolproof. Gotchas always exist. That's why we get paid the big bucks.
One way to eliminate the gotchas would be to reduce the scope of the library.
This you can do yourself by just not using this feature (serialization of pointers) I took great pains to be sure that the inclusion of this feature didn't inflict any kind of cost on those who don't use this feature.
Clearly, this is unlikely to happen, as (i) you've invested a lot of time and effort getting the pointer magic to work,
LOL - you're right - can't throw away something that works
and (ii) there are probably a lot of applications out there that would break if you did so.
Hmmm - sounds like you've answered your own question about its utility.
I for one would certainly appreciate a simplified version which omitted the pointer magic, and thus removed the const trap and similar issues. If I want to set up pointers within an object, then I can do so manually using the type-specific load()/save() functions.
This reminds me of an old joke. Man goes to a doctor and says "every time I touch my belly button it hurts." The doctor says "You mean like this" YOWWW - "yeah like that" so the doctor says "Well, don't do that!"
Obviously, I can just switch to a different serialization library (e.g. s11n).
I believe that library also has serialization of pointers
But as much as possible, I would like to stick with whatever is most "standard". And as part of the Boost family, I'm guessing that boost::serialization is top dog here. Would that be right? Is this library included in any of the proposed C++ standards (TR2, C++0x)?
This serialization library will never be part of any C++ standard.
Let's try again: I *really* appreciate the built-in support for all the STL container classes. So far, these have "just worked" perfectly for me. Nice work!
Well, Matthias Troyer has spent a fair amount of time improving that.
One tiny point. I'm aware that the "const" trap is annoying. But I've become convinced that the annoyance is small compared to the hassles it saves me and users from. I suspect that those who have a problem with aren't in the habit of using "const" everywhere they can.
It does seem bizarre to have to const-cast an object just so that it can be output. I presume that the majority of objects that are to be serialized are not intrinsically constant, so this needs to be done the majority of the time. And it looks dangerous.
The objects being serialized can't be changed during the serialization process unless tracking is turned off. If one is tracking and thereby saving only one copy of the object, and changing it in the middle tracking won't work. Because normally one doesn't want to change data while its being saved one will typically have something like the following in his program. FileSave(const MyData & md){ // create archive oa << md; // no need for "const cast" because md is already "const" ... } Now if you find that you have to use a "const cast" you should really think about why the data you're trying to save can be changing while its being saved. If saving data changes it as a side effect - you've most likely got a conceptual error in the program design. Using your approach - give me an example where avoid the "const" trap is a hard to do or makes the code ugly and I'll show you a program that is fundamentally mis-concieved.
What does the const-cast actually mean in this context? I assume it means: "I promise that no other thread is going to modify the object (or anything its pointers point to) while the serialize() operation is pending. Is that correct?
Close - but no cigar. "const" cannot guarentee that other threads don't modify it. But it can guarentee that the process of serialization itself doesn't do it. That is it's intent. Robert Ramey
But if you could provide me an example of when pointer serialization > > shines, > > then I would say "Aha! Now I understand why Robert went to so much > > trouble to make this work.", and I could be at peace with the world again! :-)
LOL - I haven't had time to make any real apps lately.
So... no example at all?
My problem is that I just don't see when pointer serialization would be a big win. But I can't tell if I just have a limited perspective. That's why I was genuinely hoping that you could provide a concrete example.
I have a tree like structure, where nodes are connected using std::vectors of boost::shared_ptr's, and I also have a cache mechanism in the root which also points to individual leafs in the tree using a std::map. This boost::serialises without any effort on my part (aside from writing the basic serialise functions for each node type where I just serialise the vector/map/members and everything else is done for me!!) Is that a good example? Certainly would be a very common scenario, or am I completely missing the point somewhere? James This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.
Dominick Layfield wrote:
"Robert Ramey" <ramey@rrsd.com> wrote:
Actually, I've considered reviewing all the email in this list and summarizing the results in a section "Hints and Troubleshooting Tips". I think this is useful and necessary and useful, but I just havent gotten around to it.
That sounds ideal.
But if you could provide me an example of when pointer serialization shines, then I would say "Aha! Now I understand why Robert went to so much trouble to make this work.", and I could be at peace with the world again! :-)
LOL - I haven't had time to make any real apps lately.
So... no example at all?
My problem is that I just don't see when pointer serialization would be a big win. But I can't tell if I just have a limited perspective. That's why I was genuinely hoping that you could provide a concrete example.
See http://www.adi.com/products_sim_ad_de.htm. This product maintains numerous connections between model ports. Among numerous other under-the-hood pointers, there are containers of connections which are pairs of boost::shared_ptr<Port>'s. The state of the entire system is saved/loaded to/from file using boost::serialization. This would be undoable without the ability to serialize pointers, and such a facility in my mind could not be called a serialization library. I'm not sure if this is concrete enough for you. But I for the life of me can not see how you could serialize the state of any moderate size system without serializing pointers. Do your applications have no sharing relations? Do they not contain any polymorphic collections? Do they only create items on the stack or in static memory? Jeff Flinn
Jeff Flinn wrote:
Dominick Layfield wrote:
My problem is that I just don't see when pointer serialization would be a big win. But I can't tell if I just have a limited perspective. That's why I was genuinely hoping that you could provide a concrete example.
See http://www.adi.com/products_sim_ad_de.htm. This product maintains numerous connections between model ports. Among numerous other under-the-hood pointers, there are containers of connections which are pairs of boost::shared_ptr<Port>'s. The state of the entire system is saved/loaded to/from file using boost::serialization. This would be undoable without the ability to serialize pointers, and such a facility in my mind could not be called a serialization library.
You are the second person to cite an example of serializing shared_ptr. The question is about raw pointers.
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Peter Dimov Sent: 21 August 2007 14:28 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Serialization newbie needs help
Jeff Flinn wrote:
Dominick Layfield wrote:
My problem is that I just don't see when pointer serialization would be a big win. But I can't tell if I just have a limited perspective. That's why I was genuinely hoping that you could provide a concrete example.
See http://www.adi.com/products_sim_ad_de.htm. This product maintains numerous connections between model ports. Among numerous other under-the-hood pointers, there are containers of connections which are pairs of boost::shared_ptr<Port>'s. The state of the entire system is saved/loaded to/from file using boost::serialization. This would be undoable without the ability to serialize pointers, and such a facility in my mind could not be called a serialization library.
You are the second person to cite an example of serializing shared_ptr. The question is about raw pointers.
Sorry, been using smart pointers for too long, and forgotten there are those other ones....but surely the same principles apply? This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.
Hughes, James:
You are the second person to cite an example of serializing shared_ptr. The question is about raw pointers.
Sorry, been using smart pointers for too long, and forgotten there are those other ones....but surely the same principles apply?
The difference between a raw pointer and a smart pointer is that the type of the smart pointer gives you enough context to let you decide what objects to track, if any. auto_ptr<X> and unique_ptr<X> for example need no tracking. Reference-counted smart pointers such as shared_ptr<X> and intrusive_ptr<X> need only track their targets, not all objects of type X. They also don't have to guess whether a type needs tracking or not; if one serializes a shared_ptr<int>, that int must be tracked. (The full capabilities of shared_ptr make things much more interesting, but even the simplistic approach that assumes no user-defined deleters covers the majority of the use cases.) A raw pointer carries no extra semantic information and can point to literally _anything_: - An object that has been serialized; - ... including an element of a collection that has been serialized; - ... including a subobject; - A persistent object that is not part of the archive; - ... including a static object; - ... including an element of a collection initialized on startup; - ... including a function; - An array element; - ... or past the end of the last array element; - An object on the heap that is not owned by this pointer; - An object on the heap that is owned by this pointer; and so on. The serialization library tries to cover some of the most common uses. Whether supporting the last bullet is a feature or a misfeature is a matter of opinion; I personally would never need this capability in user code (except as an implementation detail in the serialization of intrusive_ptr.) On the other hand, I've encountered several uses for the ability to serialize a pointer to a static object or a function, something that boost::serialization doesn't currently support. I'm not sure what the "best" serialization system must do when handed a pointer, but it's certainly possible to design a value-based system that does no tracking on its own. Robert Ramey:
Hmmm - it wouldn't occur to me that one could serialize a shared_ptr without the ability to serialize a raw pointer.
Correct shared_ptr serialization requires a duplicate tracking infrastructure anyway; it's possible to partially reuse the built-in tracking, but it's also possible to not use it at all.
Peter Dimov wrote:
You are the second person to cite an example of serializing shared_ptr. The question is about raw pointers.
Hmmm - it wouldn't occur to me that one could serialize a shared_ptr without the ability to serialize a raw pointer. A shared pointer is just a wrapper around a raw pointer and serialization of some data structure requires serialization of its components. Robert Ramey
participants (1)
-
Dominick Layfield -
Hughes, James -
Jeff Flinn -
Peter Dimov -
Robert Ramey