Hi guys, I'm wondering whether you know a solution to the following problem: I wish to have a tcp connection that sends binary messages of dynamic length. I plan on using asio, but I am concerned with the following: say i send 5 messages with lengths (10, 5, 30, 2, 3). As far as I understand TCP, when I read the stream on the other end, I will not receive the data 1 message at a time. Instead I will receive a stream that may hold N messages and may even hold portions of a message. eg the stream may hold 13 bytes, which is the first message and 3 bites of the 2nd. Are there any solutions to this? ie building messages as they come in through the TCP stream. I know how to implement this myself however I assume that this is probably a common problem. Thanks.
bringiton bringiton wrote:
Are there any solutions to this? ie building messages as they come in through the TCP stream.
Parse data from a socket stream the same way you'd parse data from any other sort of stream, such as a file. That is, design the protocol such that end-points can parse the message using its own contents, rather than meta-data such as 'messages.' Possibilities: 1) Use delimiters. A. Use text and use CRLF for a delimiter. B. Use a special octet as a delimiter, such as the null character. 2) Use message content-length
Thanks for tip aaron. I'm planning on using a message length byte at the start of the packet. Therefore the reader reads the byte and then waits until there are X bytes in the stream before it reads. I asssume you are talking about << and >> operators for parsing. My concern here is that if the message has not been completely sent, then the reader will be blocked. I'm using TCP asynchronously. On 5/22/07, Aaron W. LaFramboise <aaronrabiddog51@aaronwl.com> wrote:
bringiton bringiton wrote:
Are there any solutions to this? ie building messages as they come in through the TCP stream.
Parse data from a socket stream the same way you'd parse data from any other sort of stream, such as a file. That is, design the protocol such that end-points can parse the message using its own contents, rather than meta-data such as 'messages.'
Possibilities: 1) Use delimiters. A. Use text and use CRLF for a delimiter. B. Use a special octet as a delimiter, such as the null character. 2) Use message content-length _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Hello, My 2c: I've found that the easiest way to do this is to use blocking reads. I'm not sure whether you can get any performance benefit by not using blocking reads; I kind of doubt it. Also using blocking reads simplifies things _a lot_, since all you have to do is: 1) read a known number of bytes that tells you how many more you have to read 2) read as many bytes as resulted at step 1) I'd reserve at least the first two bytes of the message for the size - you may want to send messages larger than 256 bytes. Since the requirement is for binary messages, from Aaron's email 1.A is not applicable, while 1.B complicates things needlessly, having to use escape characters and such. Hope this helps, Radu On 5/22/07, bringiton bringiton <kneeride@gmail.com> wrote:
Thanks for tip aaron.
I'm planning on using a message length byte at the start of the packet. Therefore the reader reads the byte and then waits until there are X bytes in the stream before it reads.
I asssume you are talking about << and >> operators for parsing. My concern here is that if the message has not been completely sent, then the reader will be blocked. I'm using TCP asynchronously.
On 5/22/07, Aaron W. LaFramboise <aaronrabiddog51@aaronwl.com> wrote:
bringiton bringiton wrote:
Are there any solutions to this? ie building messages as they come in through the TCP stream.
Parse data from a socket stream the same way you'd parse data from any other sort of stream, such as a file. That is, design the protocol such that end-points can parse the message using its own contents, rather than meta-data such as 'messages.'
Possibilities: 1) Use delimiters. A. Use text and use CRLF for a delimiter. B. Use a special octet as a delimiter, such as the null character. 2) Use message content-length _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On Tue, 22 May 2007 15:52:58 +0200 "Radu-Adrian Popescu" <rpopescu@gmail.com> wrote:
My 2c: I've found that the easiest way to do this is to use blocking reads. I'm not sure whether you can get any performance benefit by not using blocking reads; I kind of doubt it. Also using blocking reads simplifies things _a lot_, since all you have to do is: 1) read a known number of bytes that tells you how many more you have to read 2) read as many bytes as resulted at step 1)
With Asio (I'm looking at 0.3.8rc3), async reads for this (common) form of "message protocol" is just as easy as doing blocking reads. Specifically, the "asio::async_read" functions specify a "completion_condition", which defaults to the protocol mentioned above. (I like the "completion_condition" design, since it allows an easy override of message boundary logic, while defaulting to the common cases.) Even if Asio didn't provide nice default handling of "message boundaries", it's pretty easy to write the logic yourself in a non-blocking design, whether using an async (proactive) or reactive model. And I think you *would* get performance benefits using the async_read capabilities, or at the very least concurrency benefits, specially with large messages. As a real-world use case, I wrote a networking infrastructure library for a project that typically sent messages between 1K and 8K bytes in length, but at certain times would send messages dozens (or hundreds) of megabytes long. Having the application (or a thread within the app) block while collecting all of the data for a single message was unacceptable. The library multiplexed all network reads and writes transparently and concurrently for the application (across multiple connections). Note that the "concurrency" I'm mentioning here is not threading concurrency, but IO multiplexing (between multiple sockets / connections) concurrency (it had the same general event multiplexing model as Asio does). Cliff
On 5/22/07, Cliff Green <cliffg@codewrangler.net> wrote:
Even if Asio didn't provide nice default handling of "message boundaries", it's pretty easy to write the logic yourself in a non-blocking design, whether using an async (proactive) or reactive model.
I'd say that its not asio's job to provide any messaging semantics, and I applaud Chris for keeping the library focused on its core goals. Higher level functionality can come later, but you have all the tools you need to do this in asio already. I personally find that having to implement a "connection" class for a given communications protocol encourages good design. You end up breaking the protocol down into small functions that are easy to read and debug. Your class ends up being a nice little FSM, which is the Right Way to do this sort of thing anyway. Phooey to blocking reads. -- Caleb Epstein
Although I do somewhat agree with Cliff, I find the phooey argumentation even better. I've read some papers that basically stated that using a thread based model with blocking IO is equivalent with using asynchronous IO. Performance of either implementation is something that definitely depends on the underlying OS and the author's skills, however common knowledge teaches us that the asynchronous IO approach is more performant, most likely due to deficiencies of the OS's thread implementation. I've written several applications using either approach, for the async style using Mina (Java) and ACE, and although I've been quite pleased with the turnout, I do find that a threading/blocking based approach is much easier to understand; but then again maybe I'm just a little bit thick. Oh and !phooey to the completion_condition design, it's excellent! On 5/23/07, Caleb Epstein <caleb.epstein@gmail.com> wrote:
On 5/22/07, Cliff Green <cliffg@codewrangler.net> wrote:
Even if Asio didn't provide nice default handling of "message boundaries", it's pretty easy to write the logic yourself in a non-blocking design, whether using an async (proactive) or reactive model.
I'd say that its not asio's job to provide any messaging semantics, and I applaud Chris for keeping the library focused on its core goals. Higher level functionality can come later, but you have all the tools you need to do this in asio already.
I personally find that having to implement a "connection" class for a given communications protocol encourages good design. You end up breaking the protocol down into small functions that are easy to read and debug. Your class ends up being a nice little FSM, which is the Right Way to do this sort of thing anyway.
Phooey to blocking reads.
-- Caleb Epstein _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Radu-Adrian Popescu Front Office Software Engineer Optiver Holding B.V. De Ruyterkade 112 1011 AB Amsterdam Office +31-(0)20-531-9000 Mobile +40 723 33 55 73
On 5/22/07, bringiton bringiton <kneeride@gmail.com> wrote:
Thanks for tip aaron.
I'm planning on using a message length byte at the start of the packet. Therefore the reader reads the byte and then waits until there are X bytes in the stream before it reads.
Check out the Chat example in the Asio docs, it covers exactly this sort of scenario. Richard
I asssume you are talking about << and >> operators for parsing. My concern here is that if the message has not been completely sent, then the reader will be blocked. I'm using TCP asynchronously.
On 5/22/07, Aaron W. LaFramboise <aaronrabiddog51@aaronwl.com> wrote:
bringiton bringiton wrote:
Are there any solutions to this? ie building messages as they come in through the TCP stream.
Parse data from a socket stream the same way you'd parse data from any other sort of stream, such as a file. That is, design the protocol such that end-points can parse the message using its own contents, rather than meta-data such as 'messages.'
Possibilities: 1) Use delimiters. A. Use text and use CRLF for a delimiter. B. Use a special octet as a delimiter, such as the null character. 2) Use message content-length _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
participants (2)
-
Aaron W. LaFramboise -
bringiton bringiton -
Caleb Epstein -
Cliff Green -
Radu-Adrian Popescu -
Richard Dingwall