Re: [boost] Boost.Buffers: network-agnostic buffer types and algorithms

wt., 7 lut 2023 o 00:31 Vinnie Falco via Boost napisał(a):

On Mon, Feb 6, 2023 at 2:43 PM Gavin Lambert via Boost wrote:

...

Perhaps, though as I said I find the "optional is more fundamental" argument more compelling, so by that logic the compatibility code should be in serialization and not optional.

I just want to underscore that the question of "where does optional go" is completely orthogonal to this proposed Boost.Buffers library, and the buffers::source and buffers::sink interfaces in particular. My proposed library has nothing to do with serialization and everything to do with providing buffer-oriented interfaces.

In fact, buffers::source is more fundamental than serialization. Boost.Serialization sits between buffers::source and the algorithm for serialization of an optional. This diagram shows the relationship:

{ host } <---> Boost.Buffers <---> Boost.Serialization <---> { optional }

In this diagram "host" is the host program or library which is implemented in terms of Boost.Buffer's abstract interfaces (buffers::source for the case of using Boost.Serialization). As seen in the diagram, the question of where the serialization spec for "optional" goes, is unrelated to Boost.Buffers.

Another way to think of it is like this. If Boost.Serialization implements the buffers::source interface, then any host which uses Boost.Buffers will be able to make use of types archived through Boost.Serialization. I'm not saying whether this is particularly useful or not as that would depend on the specific use-case. For the upcoming Boost.Http.Proto library, this means that anything which defines a serialization for Boost.Serialization could be attached as the body of an HTTP message.

The diagram for Boost.JSON looks different:

{ host } <---> Boost.Buffers <---> Boost.JSON

Here, if Boost.JSON wraps its existing streaming serializer for the buffers::source interface then any Boost.Buffers-enabled host can make use of JSON sources without depending on the Boost.JSON library. There's a network effect at play; every library that enables Boost.Buffers geometrically increases the utility of every other library that uses Boost.Buffers. Furthermore, this scales to any number of libraries and resolves the question of where the buffers::sink (or buffers::source) algorithm goes. It goes in the library which already implements a buffer-oriented data type. This includes but is not limited to:

* JSON * XML * Boost.Serialization archives * Mustache templates and output

As indicated earlier, I support the extraction of Boost.Buffers into a separate Boost library. Let me offer some further thoughts. 1. It is more of a question. These buffers::sink and buffers::source being mentioned in this thread: are they already present in one of the Boost libraries (ASIO, Beast)? 2. For any "vocabulary interface" we could separate out three parts: (1) the interface, (2) the sinks, (3) the sources. I am slightly abusing the terms, so let me give an example of the STL. Iterators are the interface. The algorithms are "sinks": they do not have to know about containers. The containers are the "sources" they do not have to know about iterators. If the STL was a Boost library, the only thing that other Boost libraries would depend on would be iterators -- not the algorithms, not the containers -- because it is only the iterators -- the interface -- that lets you plug your library into the system. I see the same situation with the proposed Boost.Buffers. The only thing that multiple libraries (Boost and non-Boost) would benefit from is the buffer interface (boost::sink, boost::source). No buffer implementations, no buffer algorithms. As you mentioned earlier, things like a circular buffer are shared between at least two libraries, so there is a merit in separating this, but this is just an engineering trade-off, rather than a vocabulary interface design (like boost::sink, boost::source). This remark applies to every library with a sound vocabulary-like interface: the interface should go separately. But that would be a very small library, maybe not worth a separate infrastructure. 3. There are a number of interfaces, where everyone can plug their type, in the STD and Boost that have an overlap. We have the IOStream interface, we have Boost.Serialization interface, std::format is coming, and we have now Boost.Buffer being proposed. Can you make a clear distinction why Boost.Buffers is different? Why do we need another one? Are previous ones defective (and can be superseded), or do they play a different, incompatible role? I suppose that buffers have much to do with buffering, that is working with chunks of messages. But how does this work with JSON? Can you even start thinking about parsing JSON if you only have a part, broken at an arbitrary position? Also, IOStreams have a layer of buffers. What is the relation there (Between IOStream buffers and the proposed Boost.Buffers)? 4. Would Boost.Buffers satisfy my every use case for buffers? Are buffers only about how you allocate a new chunk of memory? Or are they about identifing a place where you can cut your message into meaningful portions? Regards, &rzej;