[iostreams] How can I use the Stream Insertion operator (<<) with a filtering_streambuf?
Partially asked in another question, but so far unanswered: Once I create a filtering_streambuf that is connected to the zlib_compressor() and the output file, how do I then construct an object that can accept data from the Stream Insertion Operator? void CompressUsingFilteringStreambuf() { std::ofstream myFile("hello.z", std::ios_base::out | std::ios_base::binary); boost::iostreams::filtering_streambuf<boost::iostreams::output> out; out.push(boost::iostreams::zlib_compressor()); out.push(myFile); // MY QUESTION: How can I use the "stream insertion" operator (<<) // to go through the filtering_streambuf // and place compressed text into myFile? // HOW DO I CREATE someStream? someStream << "text I want to be compressed and end up in the file..."; myFile.close(); } I have scoured the documentation and the web looking for the answer -- and come up empty. Stephen
On 07/25/13 19:37, Stephen Greenfield wrote:
Partially asked in another question, but so far unanswered: Once I create a filtering_streambuf that is connected to the zlib_compressor() and the output file, how do I then construct an object that can accept data from the Stream Insertion Operator?
void CompressUsingFilteringStreambuf() { std::ofstream myFile("hello.z", std::ios_base::out | std::ios_base::binary);
boost::iostreams::filtering_streambuf<boost::iostreams::output> out; out.push(boost::iostreams::zlib_compressor()); out.push(myFile);
// MY QUESTION: How can I use the "stream insertion" operator (<<) // to go through the filtering_streambuf // and place compressed text into myFile?
// HOW DO I CREATE someStream? someStream << "text I want to be compressed and end up in the Instead of someStream, try myFile, and see if the resulting hello.z is what you want. file...";
myFile.close(); }
I have scoured the documentation and the web looking for the answer -- and come up empty.
Stephen
On 7/25/13 8:22 PM, "Larry Evans" <cppljevans@suddenlink.net<mailto:cppljevans@suddenlink.net>> wrote:
Instead of someStream, try myFile, and see if the resulting hello.z is what you want.
Larry -- I should have mentioned: I TRIED exactly that: myFile << "some text I want to be compressed"; And in that case, the text DOES end up in the file — but UNCOMPRESSED. Apparently, that bypasses the filtering_streambuf. I'm beginning to wonder if 1) it's not possible, or 2) I should try to contact the original authors of that class for their insight. I'm getting the sense this isn't one of the more commonly-understood areas of of the Boost libraries. One additional note: While I am linking in the ZLIB libraries and including their headers, I don't believe I have ANY Boost binaries linked in — only headers that are relevant. But — I'm compiling, linking and executing… Stephen -- On 07/25/13 19:37, Stephen Greenfield wrote: Partially asked in another question, but so far unanswered: Once I create a filtering_streambuf that is connected to the zlib_compressor() and the output file, how do I then construct an object that can accept data from the Stream Insertion Operator? void CompressUsingFilteringStreambuf() { std::ofstream myFile("hello.z", std::ios_base::out | std::ios_base::binary); boost::iostreams::filtering_streambuf<boost::iostreams::output> out; out.push(boost::iostreams::zlib_compressor()); out.push(myFile); // MY QUESTION: How can I use the "stream insertion" operator (<<) // to go through the filtering_streambuf // and place compressed text into myFile? // HOW DO I CREATE someStream? someStream << "text I want to be compressed and end up in the file..."; myFile.close(); } I have scoured the documentation and the web looking for the answer -- and come up empty. Stephen _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org<mailto:Boost-users@lists.boost.org> http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 26/07/2013 01:37, Stephen Greenfield wrote:
Partially asked in another question, but so far unanswered: Once I create a filtering_streambuf that is connected to the zlib_compressor() and the output file, how do I then construct an object that can accept data from the Stream Insertion Operator?
void CompressUsingFilteringStreambuf() { std::ofstream myFile("hello.z", std::ios_base::out | std::ios_base::binary);
boost::iostreams::filtering_streambuf<boost::iostreams::output> out; out.push(boost::iostreams::zlib_compressor()); out.push(myFile);
// MY QUESTION: How can I use the "stream insertion" operator (<<) // to go through the filtering_streambuf // and place compressed text into myFile?
// HOW DO I CREATE someStream? someStream << "text I want to be compressed and end up in the file...";
myFile.close(); }
I have scoured the documentation and the web looking for the answer -- and come up empty.
Stephen
Try: std::ofstream myFile("hello.z", std::ios_base::out | std::ios_base::binary); boost::iostreams::filtering_streambuf<boost::iostreams::output> out; out.push(boost::iostreams::zlib_compressor()); out.push(myFile); std::ostream os(&out); os << "text I want to be compressed and end up in the file..." On input it would be something like: std::ifstream myFile( "hello.z", std::ios::in | std::ios::binary ); boost::iostreams::filtering_streambuf< boost::iostreams::input > in; in.push( boost::iostreams::gzip_decompressor() ); in.push( ifs ); std::istream is( &in ); is >> text; Ken Appleby Soft Optics Ltd
Ken -- Much thanks for your response. I DID try that very syntax -- but in that case NOTHING wound up in the file -- it was empty. I think I also tried to flush the ostream -- but that didn't work. There seems to be some magic here I am missing -- or maybe it's a problem with the GNU 4.0 G++ compiler/libraries, or an Apple / Mac OS specific implementation issue... Stephen On Jul 25, 2013, at 11:10 PM, "Ken Appleby" <ken.appleby@btopenworld.com> wrote:
Try:
std::ofstream myFile("hello.z", std::ios_base::out | std::ios_base::binary); boost::iostreams::filtering_streambuf<boost::iostreams::output> out; out.push(boost::iostreams::zlib_compressor()); out.push(myFile);
std::ostream os(&out); os << "text I want to be compressed and end up in the file..."
On input it would be something like:
std::ifstream myFile( "hello.z", std::ios::in | std::ios::binary ); boost::iostreams::filtering_streambuf< boost::iostreams::input > in; in.push( boost::iostreams::gzip_decompressor() ); in.push( ifs ); std::istream is( &in ); is >> text;
Ken Appleby
Soft Optics Ltd
This is how I do it with file_sink: boost::iostreams::filtering_ostream output_file; output_file.push(boost::iostreams::bzip2_compressor(boost::iostreams::bzip2_params(9))); output_file.push(boost::iostreams::file_sink("file.bz2", std::ios_base::binary)); output_file<<"blalba"; I think if you do just this, nothing happen until output_file is destroyed. So enclose in { }. Frédéric
On 07/26/13 06:39, Frédéric Bron wrote:
This is how I do it with file_sink:
boost::iostreams::filtering_ostream output_file; output_file.push(boost::iostreams::bzip2_compressor(boost::iostreams::bzip2_params(9))); output_file.push(boost::iostreams::file_sink("file.bz2", std::ios_base::binary)); output_file<<"blalba";
I think if you do just this, nothing happen until output_file is destroyed. So enclose in { }.
Frédéric
Thanks Frederic. However, I think this is not obvious. I would have thought Ken's solution would have worked. I would have also thought mine would have worked until I saw Ken's which indicated std::ios_base::binary was needed when the std::ofstream was created. Is there anywhere in the docs explaining why your method works and Ken's doesn't? -regards, Larry
On 26/07/2013 14:49, Larry Evans wrote:
On 07/26/13 06:39, Frédéric Bron wrote:
This is how I do it with file_sink:
boost::iostreams::filtering_ostream output_file; output_file.push(boost::iostreams::bzip2_compressor(boost::iostreams::bzip2_params(9))); output_file.push(boost::iostreams::file_sink("file.bz2", std::ios_base::binary)); output_file<<"blalba";
I think if you do just this, nothing happen until output_file is destroyed. So enclose in { }.
Frédéric
Thanks Frederic.
However, I think this is not obvious. I would have thought Ken's solution would have worked. I would have also thought mine would have worked until I saw Ken's which indicated std::ios_base::binary was needed when the std::ofstream was created.
Is there anywhere in the docs explaining why your method works and Ken's doesn't?
-regards, Larry
Larry, Passing the filtering_streambuf to a stream constructor should indeed work, as you expect. The test program below works fine for me - boost 1_49 on mingw 4.4. Ken Appleby Soft Optics Ltd ------------------- #include <iostream> #include <fstream> #include <boost/iostreams/filtering_streambuf.hpp> #include <boost/iostreams/copy.hpp> #include <boost/iostreams/filter/gzip.hpp> int main() { try { { std::ofstream myFile("hello.gz", std::ios_base::out | std::ios_base::binary); boost::iostreams::filtering_streambuf<boost::iostreams::output> out; out.push(boost::iostreams::gzip_compressor()); out.push( myFile ); std::ostream os(&out); os << "text I want to be compressed and end up in the file..."; } { std::ifstream myFile( "hello.gz", std::ios::in | std::ios::binary ); boost::iostreams::filtering_streambuf< boost::iostreams::input > in; in.push( boost::iostreams::gzip_decompressor() ); in.push( myFile ); std::istream is( &in ); std::string text; while ( is.good() ) { char ch = is.get(); if ( is.good() ) text.push_back( ch ); } std::cout << "text from file is \"" << text << "\"" << std::endl << std::flush; } } catch ( std::exception& e ) { std::cerr << e.what() << std::endl; } }
SUCCESS! A combination of hints from all who replied made the difference: 1) Frédéric pointed out that "nothing happen[s] until output_file [filtering_ostream] is destroyed. So enclose in { }". This was also key, because I was trying to do file.close() on my ofstream BEFORE my filtering_streambuf was destroyed. That explains why the file was empty! Re-reading the documentation reveals: "By default, if the Device at the end of the chain is popped or if the filtering_stream is complete when it is destroyed, all the filters and devices in the chain are closed using the function close. This behavior can be modified using the member function set_auto_close" This states is there is no need to "pop" the compressor() or the ofstream off the filtering_stream's stack, nor to call close(). Just destruct the filtering_stream object, and everything gets written out and cleaned up. 2) Ken Appleby's source code was very complete, and allowed me to confirm the proper sequence of creating the ostream from the filtering_streambuf. 3) Holger Gerth questioned why I was using filtering_streambuf when I could've been using filtering_stream. Truth is, I wasn't sure, however, in my experiments I could neither construct the ostream (which I required to pass to other functions) from the filtering_stream, nor could I pass the filtering_stream in place of the ostream I required. Even after reading several articles on filtering_streambuf vs filtering_stream, I'm still mystified how and why (FOR MY PURPOSE) I would use the filtering_stream over constructing an ostream from a filtering_streambuf. Thanks for all the help -- I REALLY could not have done it without your collective wisdom. Stephen
On 07/26/13 21:26, Stephen Greenfield wrote: [snip]
3) Holger Gerth questioned why I was using filtering_streambuf when I could've been using filtering_stream. Truth is, I wasn't sure, however, in my experiments I could neither construct the ostream (which I required to pass to other functions) from the filtering_stream, nor could I pass the filtering_stream in place of the ostream I required.
Even after reading several articles on filtering_streambuf vs filtering_stream, I'm still mystified how and why (FOR MY PURPOSE) I would use the filtering_stream over constructing an ostream from a filtering_streambuf.
[snip] Hi Stephen, I thought the filtering_stream would be derived from std::ostream, at least according to: Derived class of std::basic_istream, std::basic_ostream or std::basic_iostream, used to perform filtering. quoted from: http://www.boost.org/doc/libs/1_54_0/libs/iostreams/doc/classes/filtering_st... Did you not find this so, or do you needs require something more specific than std::ostream& to be passed to the other functions you mention? -regards, Larry
On 7/28/13 8:45 PM, "Larry Evans" <cppljevans@suddenlink.net<mailto:cppljevans@suddenlink.net>> wrote: I thought the filtering_stream would be derived from std::ostream, at least according to: Derived class of std::basic_istream, std::basic_ostream or std::basic_iostream, used to perform filtering. quoted from: http://www.boost.org/doc/libs/1_54_0/libs/iostreams/doc/classes/filtering_st... Did you not find this so, or do you needs require something more specific than std::ostream& to be passed to the other functions you mention? Larry -- Thanks! You would've thought that — and you know what? You would be CORRECT. I thought I had tested every possible case — but I think I did that BEFORE discovering my error with closing the file before destructing the filtering_streambuf/filtering_ostream. You (and everyone else) were ABSOLUTELY CORRECT. When I changed my code from a filtering_streambuf: ofstream myXMLFile (destPath, std::ios_base::out | std::ios_base::app | std::ios_base::binary); boost::iostreams::filtering_streambuf<boost::iostreams::output> filteringOStream; filteringOStream.push(boost::iostreams::zlib_compressor()); filteringOStream.push(myXMLFile); std::ostream myExportedXMLFilteringOStream(&filteringOStream); Foo(myExportedXMLFilteringOStream); TO A filtering_ostream: ofstream myXMLFile (destPath, std::ios_base::out | std::ios_base::app | std::ios_base::binary); boost::iostreams::filtering_ostream myExportedXMLFilteringOStream; myExportedXMLFilteringOStream.push(boost::iostreams::zlib_compressor()); myExportedXMLFilteringOStream.push(myXMLFile); Foo(myExportedXMLFilteringOStream); Thank you so much for tying up that loose end! I'm assuming the filtering_ostream is of course more efficient than the filtering_streambuf scenario? Best Regards, Stephen
On 07/28/13 23:28, Stephen Greenfield wrote:
On 7/28/13 8:45 PM, "Larry Evans" <cppljevans@suddenlink.net <mailto:cppljevans@suddenlink.net>> wrote:
I thought the filtering_stream would be derived from std::ostream, at least according to:
Derived class of std::basic_istream, std::basic_ostream or std::basic_iostream, used to perform filtering.
quoted from:
http://www.boost.org/doc/libs/1_54_0/libs/iostreams/doc/classes/filtering_st...
Did you not find this so, or do you needs require something more specific than std::ostream& to be passed to the other functions you mention?
Larry --
Thanks! You would've thought that — and you know what? You would be CORRECT.
I thought I had tested every possible case — but I think I did that BEFORE discovering my error with closing the file before destructing the filtering_streambuf/filtering_ostream.
You (and everyone else) were ABSOLUTELY CORRECT. When I changed my code from a filtering_streambuf:
ofstream myXMLFile (destPath, std::ios_base::out | std::ios_base::app | std::ios_base::binary); boost::iostreams::filtering_streambuf<boost::iostreams::output> filteringOStream; filteringOStream.push(boost::iostreams::zlib_compressor()); filteringOStream.push(myXMLFile); std::ostream myExportedXMLFilteringOStream(&filteringOStream); Foo(myExportedXMLFilteringOStream);
TO A filtering_ostream:
ofstream myXMLFile (destPath, std::ios_base::out | std::ios_base::app | std::ios_base::binary); boost::iostreams::filtering_ostream myExportedXMLFilteringOStream; myExportedXMLFilteringOStream.push(boost::iostreams::zlib_compressor()); myExportedXMLFilteringOStream.push(myXMLFile); Foo(myExportedXMLFilteringOStream);
Thank you so much for tying up that loose end! I'm assuming the filtering_ostream is of course more efficient than the filtering_streambuf scenario? I don't know, but I guess so.
BTW, I tried modifying Ken's code to use filtering_ostream instead of filtering_streambuf, and it worked. I also removed the ios_base flags and it still worked. Then I tried using stringstream instead of filestream and it no longer worked. Anyone have any clues about why stringstream doesn't work? The code is attached. With the attached code and attached Jamfile.v2, and with: #define USE_STRINGSTRM in the code, the KenApplebyStrm.output produced by bjam is in the next attachment. In contrast, with: //#define USE_STRINGSTRM int the code, the KenApplebyStrm.output produced by bjam is in the last attachment.
Best Regards,
Stephen
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 07/29/13 11:10, Larry Evans wrote: [snip]
BTW, I tried modifying Ken's code to use filtering_ostream instead of filtering_streambuf, and it worked. I also removed the ios_base flags and it still worked. Then I tried using stringstream instead of filestream and it no longer worked. Anyone have any clues about why stringstream doesn't work? The code is attached. With the attached code and attached Jamfile.v2, and with:
#define USE_STRINGSTRM
in the code, the KenApplebyStrm.output produced by bjam is in the next attachment. In contrast, with: //#define USE_STRINGSTRM int the code, the KenApplebyStrm.output produced by bjam is in the last attachment. [snip] OOPS. The code attached previously used streambuf instead of stream. The current attachment uses stream.
On 07/29/13 11:27, Larry Evans wrote:
On 07/29/13 11:10, Larry Evans wrote: [snip]
BTW, I tried modifying Ken's code to use filtering_ostream instead of filtering_streambuf, and it worked. I also removed the ios_base flags and it still worked. Then I tried using stringstream instead of filestream and it no longer worked. Anyone have any clues about why stringstream doesn't work? The code is attached. With the attached code and attached Jamfile.v2, and with:
#define USE_STRINGSTRM
in the code, the KenApplebyStrm.output produced by bjam is in the next attachment. In contrast, with: //#define USE_STRINGSTRM int the code, the KenApplebyStrm.output produced by bjam is in the last attachment. [snip] OOPS. The code attached previously used streambuf instead of stream. The current attachment uses stream.
Based on running the attached both with: #define USE_FILT_COMPRESS then with: //#define USE_FILT_COMPRESS on line 40 showed that using the gzip_decompressor starts reading at the end of the stream instead of at the beginning. The corresponding outputs are also attached. This contrasts with the count filter which starts reading, as one would expect, at the beginning of the stream. Still trying to track down why. -Larry
On 07/30/13 13:59, Larry Evans wrote:
On 07/29/13 11:27, Larry Evans wrote:
On 07/29/13 11:10, Larry Evans wrote: [snip]
BTW, I tried modifying Ken's code to use filtering_ostream instead of filtering_streambuf, and it worked. I also removed the ios_base flags and it still worked. Then I tried using stringstream instead of filestream and it no longer worked. Anyone have any clues about why stringstream doesn't work? The code is attached. With the attached code and attached Jamfile.v2, and with:
#define USE_STRINGSTRM
in the code, the KenApplebyStrm.output produced by bjam is in the next attachment. In contrast, with: //#define USE_STRINGSTRM int the code, the KenApplebyStrm.output produced by bjam is in the last attachment. [snip] OOPS. The code attached previously used streambuf instead of stream. The current attachment uses stream.
Based on running the attached both with:
#define USE_FILT_COMPRESS
then with:
//#define USE_FILT_COMPRESS
on line 40 showed that using the gzip_decompressor starts reading at the end of the stream instead of at the beginning.
The corresponding outputs are also attached.
This contrasts with the count filter which starts reading, as one would expect, at the beginning of the stream.
Still trying to track down why.
Changing the: out.flush() around line 93 to: out.strict_sync(); which is described here: http://www.boost.org/doc/libs/1_54_0/libs/iostreams/doc/classes/filtering_st... failed to solve problem :( Still trying to track down why. -regards, Larry
On 07/31/13 07:16, Larry Evans wrote:
On 07/30/13 13:59, Larry Evans wrote:
On 07/29/13 11:27, Larry Evans wrote:
On 07/29/13 11:10, Larry Evans wrote: [snip]
BTW, I tried modifying Ken's code to use filtering_ostream instead of filtering_streambuf, and it worked. I also removed the ios_base flags and it still worked. Then I tried using stringstream instead of filestream and it no longer worked. Anyone have any clues about why stringstream doesn't work? The code is attached. With the attached code and attached Jamfile.v2, and with:
#define USE_STRINGSTRM
in the code, the KenApplebyStrm.output produced by bjam is in the next attachment. In contrast, with: //#define USE_STRINGSTRM int the code, the KenApplebyStrm.output produced by bjam is in the last attachment. [snip] OOPS. The code attached previously used streambuf instead of stream. The current attachment uses stream.
Based on running the attached both with:
#define USE_FILT_COMPRESS
then with:
//#define USE_FILT_COMPRESS
on line 40 showed that using the gzip_decompressor starts reading at the end of the stream instead of at the beginning.
The corresponding outputs are also attached.
This contrasts with the count filter which starts reading, as one would expect, at the beginning of the stream.
Still trying to track down why.
Changing the:
out.flush()
around line 93 to:
out.strict_sync();
which is described here:
http://www.boost.org/doc/libs/1_54_0/libs/iostreams/doc/classes/filtering_st...
failed to solve problem :( Still trying to track down why.
Maybe the reason is the filter's are not flushable. However, even though the counter filter is also not flushable, the output turns out OK. Finally, I read about reset and pop: http://www.boost.org/doc/libs/1_54_0/libs/iostreams/doc/classes/filtering_st... and both worked to get proper value for actual_out in the attached. The root cause of my difficulties was my assumptions about what out.flush() or out.strict_sync() should have done, and not reading about reset or pop. However, my excuse was that flush and sync are names with pretty much standard meanings; yet, with iostreams, those meanings don't apply completely :(. Hopefully this will help others running into the same problem. -regards, Larry
participants (2)
-
Frédéric Bron -
Ken Appleby -
Larry Evans -
Stephen Greenfield