Boost Serialization: Serializing large objects
We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error. This is on a 32-bit QNX system. * Is there a limit to the size of the object that can be serialized? * Is there anything we can do to fix or work around this? Here is the code we are using: std::stringstream ss_; boost::archive::text_oarchive oa(ss_, boost::archive::no_header); oa << &object; Adlai
On 10/5/17 10:41 AM, Adlai Shawareb via Boost wrote:
We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.
This is on a 32-bit QNX system.
* Is there a limit to the size of the object that can be serialized? * Is there anything we can do to fix or work around this?
Here is the code we are using:
std::stringstream ss_; boost::archive::text_oarchive oa(ss_, boost::archive::no_header); oa << &object;
Adlai
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
There is no known limitation on file size in the boost serialization library. Robert Ramey
On 10/5/17 10:41 AM, Adlai Shawareb via Boost wrote:
We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.
This is on a 32-bit QNX system.
* Is there a limit to the size of the object that can be serialized? * Is there anything we can do to fix or work around this?
Here is the code we are using:
std::stringstream ss_; boost::archive::text_oarchive oa(ss_, boost::archive::no_header); oa << &object;
Adlai
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
suggestion: Try using a text file stream rather than a stringstream. Robert Ramey
We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.
This is on a 32-bit QNX system.
Is there a limit to the size of the object that can be serialized? Is there anything we can do to fix or work around this?
Here is the code we are using:
std::stringstream ss_; boost::archive::text_oarchive oa(ss_, boost::archive::no_header); oa << &object;
Adlai
Thanks for your help. When serializing into a stringstream, how does Boost add or allocate memory to the stringstream? Adlai
On 10/9/17 4:21 PM, Adlai Shawareb via Boost wrote:
Thanks for your help. When serializing into a stringstream, how does Boost add or allocate memory to the stringstream?
Boost doesn't do it. std::stringstream does it. It's not too hard to make your own archive which is just a giant memory buffer. I think there's an example in the documentation which does this. But I might be wrong. Robert Ramey
Boost doesn't do it. std::stringstream does it.
It's not too hard to make your own archive which is just a giant memory buffer. I think there's an example in the documentation which does this. But I might be wrong.
Sorry Robert I wasn't clear. What I meant to ask was which calls to std::stringstream does Boost use to add memory to the stringstream? I looked at the Serialization Tutorial and didn't see an example of making my own archive. If you are inclined, I would appreciate an example. Adlai
On 10/10/17 10:41 AM, Adlai Shawareb via Boost wrote:
Boost doesn't do it. std::stringstream does it.
It's not too hard to make your own archive which is just a giant memory buffer. I think there's an example in the documentation which does this. But I might be wrong.
Sorry Robert I wasn't clear. What I meant to ask was which calls to std::stringstream does Boost use to add memory to the stringstream?
as far as I know there are no such calls supported by std::stringstream. But maybe I'm wrong about that. Boost serialization uses the interface provide by std::basic_stream - no more than that.
I looked at the Serialization Tutorial and didn't see an example of making my own archive. If you are inclined, I would appreciate an example.
Adlai
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
as far as I know there are no such calls supported by std::stringstream. But maybe I'm wrong about that. Boost serialization uses the interface provide by std::basic_stream - no more than that.
Thanks Robert. Does Boost use operator<< to add data to the stringstream? If not, how is data added? I would like to know so that I can look into why using that method we get an error. Adlai
On 10/10/17 11:29 AM, Adlai Shawareb via Boost wrote:
as far as I know there are no such calls supported by std::stringstream. But maybe I'm wrong about that. Boost serialization uses the interface provide by std::basic_stream - no more than that.
Thanks Robert. Does Boost use operator<< to add data to the stringstream? If not, how is data added? I would like to know so that I can look into why using that method we get an error.
Adlai
I'm sorry, I can't give short answers to these questions. You'll have to study the documentation. Robert Ramey
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Thanks Robert. Does Boost use operator<< to add data to the stringstream? If not, how is data added? I would like to know so that I can look into why using that method we get an error.
Adlai
You say the object is 700 megs. Is that 700 megs in memory or in text? The text representation is almost certainly bigger than the in memory representation so you might just be running out of memory. Alternatively, you could just be running out of address space if stringstream requires a contiguous buffer. That's an implementation detail, but I would not be surprised if stringstream allocates a contiguous buffer, in which case you almost certainly would be bumping into a memory fragmentation issue due to the 32bit address space. If you serialize directly to a file instead of stringstream, as Robert suggested, the problem might go away. -- chris
You say the object is 700 megs. Is that 700 megs in memory or in text? The text representation is almost certainly bigger than the in memory representation so you might just be running out of memory.
Alternatively, you could just be running out of address space if stringstream requires a contiguous buffer. That's an implementation detail, but I would not be surprised if stringstream allocates a contiguous buffer, in which case you almost certainly would be bumping into a memory fragmentation issue due to the 32bit address space.
If you serialize directly to a file instead of stringstream, as Robert suggested, the problem might go away.
-- chris
Thanks for responding Chris. The unserialized data is 700 megs. You are correct, the text representation is closer to 1.2 GB. We are using extended addressing on QNX, so I haven't focused on the 32bit address space / fragmentation issue, but it can't be ruled out. The contiguous buffer issue seems likely. The questions I was asking of Robert were to look for something that we could take to QNX to get some feedback. Adlai ________________________________
On 05-10-17 19:41, Adlai Shawareb via Boost wrote:
std::stringstream ss_; boost::archive::text_oarchive oa(ss_, boost::archive::no_header); oa << &object;
If you can't write directly to a file, just use `back_insert_device` with a properly reserved container (std::string or std::vector<char> for example): http://www.boost.org/doc/libs/1_65_1/libs/iostreams/doc/classes/back_inserte...
participants (4)
-
Adlai Shawareb -
Chris Glover -
Robert Ramey -
Seth