Re: [boost] New Lib "Beast", HTTP + WebSocket protocols

On Sun, Apr 24, 2016 at 7:09 AM, Bjorn Reese wrote:

This looks like a nice library...

Thanks for the kind words!

It may be a good idea if we joined the two projects together. Would you consider such an option?

I'm open to collaboration but also cautious. Beast.HTTP was designed throughout to have a narrow interface. It offers only a universal model for the HTTP message, and functions to parse, serialize, deserialize, and send/receive on sockets. It offers both synchronous and asynchronous functionality. And it accomplishes these goals with an interface that resembles Boost.Asio as closely as possible, to eliminate the learning curve for using the library. It is this author's opinion that the more a library tries to do, the more controversial it is and the harder it will be to get through the boost review process. There's a strong need for simple free functions to send and receive HTTP messages on a socket in as few lines of code as possible. Beast.HTTP achieves this. Example code: using namespace beast::http; boost::asio::ip::tcp::socket sock(ios); ... request req({method_t::http_get, "/", 11}); req.headers.replace("Host", "boost.org:80"); req.headers.replace("User-Agent", "Beast.HTTP"); write(sock, req); Full, compiling example program here: http://vinniefalco.github.io/beast/beast/intro/example.html Remember that Beast includes both HTTP and WebSocket. The HTTP support was driven by the need of the websocket stream to perform a HTTP Upgrade handshake when initializing the session. I think a litmus test for the interface quality of a HTTP library, is the ease in which it allows an implementation to perform this websocket handshake. Here is actual code from beast::websocket::stream which synchronously performs the WebSocket HTTP Upgrade handshake: template<class NextLayer> void stream<NextLayer>:: handshake(boost::string_ref const& host, boost::string_ref const& resource, error_code& ec) { std::string key; http::write(stream_, build_request(host, resource, key), ec); if(ec) return; http::responsehttp::string_body resp; http::read(next_layer_, stream_.buffer(), resp, ec); if(ec) return; do_response(resp, key, ec); } The function above is not possible with Boost.Http in its current incarnation, as there is no client support nor is there support for synchronous operations. Based on the design choices visible at the tip of trunk (for example, requiring a stream constructed around the socket to send and receive messages) it seems unlikely that such simple operations will be possible in future incarnations of that library unless major changes in its design are made.

Last year, the Boost.Http project went through a formal Boost review. The summary [2] shows what the Boost community expects from a HTTP library.

I'll quote and address each of the points brought up in [2], the feedback from the review of Boost.Http: "Boost.Http currently only provides a server-side API, but the reviewers felt that a client-side API would be usable to more users." Beast.HTTP is completely role-agnostic, and works for building clients and servers. "There was also a recurring request for Boost.Http to be a header-only library. The HTTP parser currently used is not header-only and that is the main obstacle towards a header-only Boost.Http library." Agree 100%. Beast.HTTP also uses the NodeJS parser, and that's the only bit of code that is not header only. Thursday I started on a header-only parser, here's what we have so far (please keep in mind, this is a work in progress): https://github.com/vinniefalco/rippled/blob/my-parser/src/beast/test/http/my... This parser will not have any dependencies except the standard library so if necessary the code could be pinched for other projects. "Some of the discussion revolved around what level of abstraction would be appropriate for Boost.Http. The views ranged from wanting higher- level APIs..." Higher-level APIs are great but there is danger in offering increasing levels of abstraction. At each increase, the target audience diminishes and the chances of design choices made in the abstraction becoming inappropriate for particular use-cases goes up. No matter how strong the desire for higher level APIs, they need to be built on a foundation. Beast.HTTP provides the correct foundation; it is that which cannot be broken down further and it is that upon which everything else can be built. As such, it offers library virtue with its current feature set. "...all the way down to simply wanting a HTTP parser/generator and then leave all the socket and buffer management up to the user." By design, Beast.HTTP does not perform any buffering or socket management. Such layers can be built on top; one of the examples creates a pipelining stream, see: https://github.com/vinniefalco/Beast/blob/master/examples/http_stream.h https://github.com/vinniefalco/Beast/blob/master/examples/http_async_server.... "Boost.Http currently creates an associative array for all header fields. One reviewer explored the idea of using an incremental (push or pull) HTTP parser as part of the API to let users decide which header fields to copy and which to discard." The Beast.HTTP read algorithm allows customization of the Parser template argument (*), allowing any type that meets the requirements to be used as the implementation for parsing messages. (*) planned feature "Some reviewers also felt that HTTP/2 should be part of Boost.Http, partly because that would demonstrate the extensibility of the current design, and partly because the library would be in a stronger position to attract users if it offers more than its competitors." The IETF adopted as a goal for HTTP/2, to leave the message the same (while changing its wire format). Therefore, Beast.HTTP's message model is already HTTP/2-friendly. As for the extensibility of the design, free functions to send and receive HTTP messages are fundamentally incompatible with HTTP/2, which requires ongoing state to decompress headers and multiplex streams. And yet, we know that free functions to send and receive HTTP/1 messages are useful and sorely needed. Furthermore HTTP/2 adds features that don't make sense for HTTP/1. What would you do with the stream ID? It is not part of the message model, because it describes a transport level property (and what meaning would it have for a HTTP/1 message? or someone who is using the message object but not using any sockets at all?) What about the interface to inform the peer of a new receive window size? What about setting stream priorities/weights? How do we sort all this out? The interface for HTTP/1 should consist of a universal message model, plus functions to send and receive those messages on sockets. The interface for HTTP/2 should be a class template that wraps a socket (similar in style to beast::websocket::stream), deals in the same universal message model, and offers member functions to send and receive those messages with associated stream IDs as well as adjust HTTP/2-specific session parameters. We respectfully disagree with those reviewers who feel that a single interface should serve the needs of sending and receiving both HTTP/1 and HTTP/2 messages.

In order to address one of these expectations we have a Boost Summer of Code project this year for a HTTP parser.

Hopefully this header-only parser will be done before the summer begins. One area where Beast.HTTP could use work is in the examples. Another, bigger project would be to develop a HTTP/2 stream class template for Beast using its message model. These are areas where contributions would be most welcomed and likely the most productive use of external effort.