Increase iostream zlib filter buffer size
I am using boost serialization(binary_oarchive) with iostream zlib filter. It is working fine but its too slow(takes 10hrs). After debugging I could see the buffer size as 4KB. My data is about 600MB. How can I increase the size of buffer to say 10MB. I tried following options:- 1. passing buffer size in zlib_compressor. 2. Redefining BOOST_IOSTREAMS_DEFAULT_DEVICE_BUFFER_SIZE to 10485760 But none of above seems to work. Please guide. -- View this message in context: http://boost.2283326.n4.nabble.com/Increase-iostream-zlib-filter-buffer-size... Sent from the Boost - Users mailing list archive at Nabble.com.
On 3/14/2011 11:49 AM, rajiv82 wrote:
I am using boost serialization(binary_oarchive) with iostream zlib filter. It is working fine but its too slow(takes 10hrs). After debugging I could see the buffer size as 4KB. My data is about 600MB.
How can I increase the size of buffer to say 10MB.
I tried following options:-
1. passing buffer size in zlib_compressor. 2. Redefining BOOST_IOSTREAMS_DEFAULT_DEVICE_BUFFER_SIZE to 10485760
But none of above seems to work.
When you say "none seems to work", can you be more explicit... Are you saying that when you specify the buffer size in the constructor, for example: gzip_compressor(zl_params, 10485760) That it does not use a buffer of 10485760? ...or are you saying that it does not "speed" up your file to be compressed? I recall doing compression tests of larger size in the past (though that may have been using bzip2... which is normally slower than gzip) and recall it taking minutes and not hours... Are you sure your speed issue is related to the buffer size being used in iostreams?
Yes, using gzip_compressor(zl_params, 10485760) , doesn't increase the buffer size. I tried few things , and it worked now. Modified code: Code snippet ------------------ std::ofstream ofs(outPath.c_str() , std::ios::out|std::ios::binary|std::ios::trunc); { boost::iostreams::filtering_streambuf out; out.push(boost::iostreams::gzip_compressor(); out.push(); { boost::archive::binary_oarchive oa(out); oa << data; } } Without enclosing code in braces, it takes time with 100% cpu. After enclosing in braces, it seems to be very fast. No idea how enclosing in braces increases the speed. Any suggestion ? After solving this issue, I came across with another issue. I had written decompresser with gzip, standalone utility works fine, but when I embed same code in my multi-threaded application, filestream stuck in waiting for lock(scoped_lock). Please suggest. If you want I can post the gdb stack trace. -- View this message in context: http://boost.2283326.n4.nabble.com/Increase-iostream-zlib-filter-buffer-size... Sent from the Boost - Users mailing list archive at Nabble.com.
On 3/20/2011 2:41 AM, rajiv82 wrote:
Yes, using gzip_compressor(zl_params, 10485760) , doesn't increase the buffer size.
I tried few things , and it worked now. Modified code:
Code snippet ------------------ std::ofstream ofs(outPath.c_str() , std::ios::out|std::ios::binary|std::ios::trunc); { boost::iostreams::filtering_streambuf out;
out.push(boost::iostreams::gzip_compressor(); out.push(); { boost::archive::binary_oarchive oa(out); oa<< data; } }
Without enclosing code in braces, it takes time with 100% cpu. After enclosing in braces, it seems to be very fast. No idea how enclosing in braces increases the speed. Any suggestion ?
No suggestions here, but I am glad you found a work around. Perhaps others can comment.
After solving this issue, I came across with another issue.
I had written decompresser with gzip, standalone utility works fine, but when I embed same code in my multi-threaded application, filestream stuck in waiting for lock(scoped_lock).
Please suggest. If you want I can post the gdb stack trace.
I haven't used iostreams in a multi-threaded environment myself either (but I don't think it is thread safe by itself), but others have and observed similar behaviors... For example see: http://www.gamedev.net/topic/535559-boostiostreams-and-multiple-threads/
My BAD :( All the above problem is solved now. The issue was one of my static library was using boost 1.32 version. And all other were using 1.35 After using 1.35 for all, it worked great. Thanks all for your help. -- View this message in context: http://boost.2283326.n4.nabble.com/Increase-iostream-zlib-filter-buffer-size... Sent from the Boost - Users mailing list archive at Nabble.com.
participants (2)
-
eg -
rajiv82