Boost.Buffers: network-agnostic buffer types and algorithms
Greetings. I am asking if there is any interest in a Boost.Buffers library, which consists of the following: * ConstBuffer, MutableBuffer concepts * ConstBuffers, MutableBuffers concepts * const_buffer, mutable_buffer types * DynamicBuffer concept * `source` and `sink` abstract interfaces used to define the Source and Sink concepts * metafunctions for checking all type requirements * implementations of common buffers and dynamic buffers: - buffers_pair - array_of_buffers - circular_buffer - dynamic_string_buffer * Algorithms: - buffer_copy - buffer_size The twist is that this library has no dependency on Asio or Boost.Asio (but Asio will still recognize them thanks to some template heroics and a splash of forward declarations). Why am I proposing this? Well, let me tell you some things... I am currently working on these two libraries: Boost.Http.Proto Boost.Http.Io The Proto library implements HTTP/1 to a fuller feature set than Beast and fixes all the design flaws in Beast (but it is completely not API compatible). This library does not do any network I/O, it is expressed purely in terms of buffer concepts (i.e. depends on Boost.Buffer and not Boost.Asio). The Io library depends on Asio and the Proto library to provide the network algorithms which use Asio to implement HTTP/1. This is the equivalent HTTP functionality of what Beast offers today. Websockets is not included but when I am done with Http I will redo websockets with improvements and new features, and similarly they will be developed as two libraries: Boost.Websockets.Proto Boost.Websockets.Io My motivation for splitting libraries this way is as follows: * Each library is smaller and compiles more quickly * CI turnarounds are faster * Separation of concerns The HTTP Protocol library is designed in a way that much of an application's HTTP business logic can be expressed independently of Asio using only the constructs found in Http.Proto. This especially means fewer templates and faster compilation because Http.Proto is designed from the ground up to avoid templates whenever possible. However during the development of Http.Proto I find myself having to reach for the same buffer algorithms and containers that I developed for Beast (such as the circular buffer and the dynamic buffer). So Http.Proto has been steadily growing these things: https://github.com/CPPAlliance/http_proto/blob/a3a284f06597ad12ca6cc1782bebb... Extract this independent buffer functionality into a separate library Boost.Buffers would bring these benefits: * Again a smaller library, faster compilation * Faster CI turnarounds * Separation of concerns Additionally, algorithms can be expressed in terms of asio-independent buffer types and concepts. For example, Boost.JSON could implement the boost::buffers::sink and boost::buffers::source interfaces which would enable its use in Boost.Http.Proto and Boost.Requests (an upcoming HTTP client library from Klemens), without having to depend on either of those libraries or Asio. Is this style of development of smaller, fine grained libraries that separate concerns something that is desirable for Boost? Glad for any feedback. -- Regards, Vinnie Follow me on GitHub: https://github.com/vinniefalco
On Fri, 27 Jan 2023 at 20:35, Vinnie Falco via Boost
Greetings. I am asking if there is any interest in a Boost.Buffers
library, which consists of the following:
* ConstBuffer, MutableBuffer concepts * ConstBuffers, MutableBuffers concepts * const_buffer, mutable_buffer types * DynamicBuffer concept * `source` and `sink` abstract interfaces used to define the Source and Sink concepts * metafunctions for checking all type requirements * implementations of common buffers and dynamic buffers: - buffers_pair - array_of_buffers - circular_buffer - dynamic_string_buffer * Algorithms: - buffer_copy - buffer_size
The twist is that this library has no dependency on Asio or Boost.Asio (but Asio will still recognize them thanks to some template heroics and a splash of forward declarations). Why am I proposing this? Well, let me tell you some things...
I am currently working on these two libraries: Boost.Http.Proto Boost.Http.Io
The Proto library implements HTTP/1 to a fuller feature set than Beast and fixes all the design flaws in Beast (but it is completely not API compatible). This library does not do any network I/O, it is expressed purely in terms of buffer concepts (i.e. depends on Boost.Buffer and not Boost.Asio).
The Io library depends on Asio and the Proto library to provide the network algorithms which use Asio to implement HTTP/1. This is the equivalent HTTP functionality of what Beast offers today.
Websockets is not included but when I am done with Http I will redo websockets with improvements and new features, and similarly they will be developed as two libraries:
Boost.Websockets.Proto Boost.Websockets.Io
My motivation for splitting libraries this way is as follows:
* Each library is smaller and compiles more quickly * CI turnarounds are faster * Separation of concerns
The HTTP Protocol library is designed in a way that much of an application's HTTP business logic can be expressed independently of Asio using only the constructs found in Http.Proto. This especially means fewer templates and faster compilation because Http.Proto is designed from the ground up to avoid templates whenever possible.
However during the development of Http.Proto I find myself having to reach for the same buffer algorithms and containers that I developed for Beast (such as the circular buffer and the dynamic buffer). So Http.Proto has been steadily growing these things:
< https://github.com/CPPAlliance/http_proto/blob/a3a284f06597ad12ca6cc1782bebb...
Extract this independent buffer functionality into a separate library Boost.Buffers would bring these benefits:
* Again a smaller library, faster compilation * Faster CI turnarounds * Separation of concerns
Personally I think that the idea has merit on the basis of separation of concerns alone.
Additionally, algorithms can be expressed in terms of asio-independent buffer types and concepts. For example, Boost.JSON could implement the boost::buffers::sink and boost::buffers::source interfaces which would enable its use in Boost.Http.Proto and Boost.Requests (an upcoming HTTP client library from Klemens), without having to depend on either of those libraries or Asio.
Is this style of development of smaller, fine grained libraries that separate concerns something that is desirable for Boost? Glad for any feedback.
-- Regards, Vinnie
Follow me on GitHub: https://github.com/vinniefalco
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
I agree with Richard. The separation of concerns seems useful. Also the concrete implementation of some Asio concepts. Even other Asio concepts could benefit from concrete implementations in other libraries: executors, streams, tokens, etc... Em sex., 27 de jan. de 2023 às 17:04, Richard Hodges via Boost < boost@lists.boost.org> escreveu:
On Fri, 27 Jan 2023 at 20:35, Vinnie Falco via Boost < boost@lists.boost.org> wrote:
Greetings. I am asking if there is any interest in a Boost.Buffers
library, which consists of the following:
* ConstBuffer, MutableBuffer concepts * ConstBuffers, MutableBuffers concepts * const_buffer, mutable_buffer types * DynamicBuffer concept * `source` and `sink` abstract interfaces used to define the Source and Sink concepts * metafunctions for checking all type requirements * implementations of common buffers and dynamic buffers: - buffers_pair - array_of_buffers - circular_buffer - dynamic_string_buffer * Algorithms: - buffer_copy - buffer_size
The twist is that this library has no dependency on Asio or Boost.Asio (but Asio will still recognize them thanks to some template heroics and a splash of forward declarations). Why am I proposing this? Well, let me tell you some things...
I am currently working on these two libraries: Boost.Http.Proto Boost.Http.Io
The Proto library implements HTTP/1 to a fuller feature set than Beast and fixes all the design flaws in Beast (but it is completely not API compatible). This library does not do any network I/O, it is expressed purely in terms of buffer concepts (i.e. depends on Boost.Buffer and not Boost.Asio).
The Io library depends on Asio and the Proto library to provide the network algorithms which use Asio to implement HTTP/1. This is the equivalent HTTP functionality of what Beast offers today.
Websockets is not included but when I am done with Http I will redo websockets with improvements and new features, and similarly they will be developed as two libraries:
Boost.Websockets.Proto Boost.Websockets.Io
My motivation for splitting libraries this way is as follows:
* Each library is smaller and compiles more quickly * CI turnarounds are faster * Separation of concerns
The HTTP Protocol library is designed in a way that much of an application's HTTP business logic can be expressed independently of Asio using only the constructs found in Http.Proto. This especially means fewer templates and faster compilation because Http.Proto is designed from the ground up to avoid templates whenever possible.
However during the development of Http.Proto I find myself having to reach for the same buffer algorithms and containers that I developed for Beast (such as the circular buffer and the dynamic buffer). So Http.Proto has been steadily growing these things:
<
https://github.com/CPPAlliance/http_proto/blob/a3a284f06597ad12ca6cc1782bebb...
Extract this independent buffer functionality into a separate library Boost.Buffers would bring these benefits:
* Again a smaller library, faster compilation * Faster CI turnarounds * Separation of concerns
Personally I think that the idea has merit on the basis of separation of concerns alone.
Additionally, algorithms can be expressed in terms of asio-independent buffer types and concepts. For example, Boost.JSON could implement the boost::buffers::sink and boost::buffers::source interfaces which would enable its use in Boost.Http.Proto and Boost.Requests (an upcoming HTTP client library from Klemens), without having to depend on either of those libraries or Asio.
Is this style of development of smaller, fine grained libraries that separate concerns something that is desirable for Boost? Glad for any feedback.
-- Regards, Vinnie
Follow me on GitHub: https://github.com/vinniefalco
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- Alan Freitas https://alandefreitas.github.io/alandefreitas/ https://github.com/alandefreitas
On Fri, Jan 27, 2023 at 1:11 PM Alan de Freitas via Boost
Even other Asio concepts could benefit from concrete implementations in other libraries: executors, streams, tokens, etc...
Thanks... but, slow down there cowboy :) I'm only proposing buffers, and this library would not require any changes to Asio. The executor, stream, and token concepts from Asio are specific to asynchrony so I am skeptical that there is a lot of value in expressing them in their own library (and in fact I think they all need each other to be useful, i.e. an executor without an io_context is useless, as is a stream or completion token). The main purpose of Boost.Buffers is to have buffer abstractions that are independent of Asio. Thanks
As the author of an I/O runtime, I agree that there's tremendous value in an abstraction such as this so libraries and applications can better interoperate with each other using a well-defined vocabulary. Some things to consider in the design space are already existing libraries and APIs like readv/writev and libssl's BIO pairs. Whatever abstractions we so choose, they should be designed with these kinds of usages in mind first and foremost. - Christian
I do like reducing dependencies, but I don't see the point of separating things just for the sake of it. If you're writing a boost library most users will have installed boost as a monolith; but even those that didn't will use this in the context of http+asio so that an avoiding the inclusion of `asio/buffer.hpp` doesn't really seem like a great motivation for a new library to me. I for one don't mind using sub-libraries, e.g. including boost/core for the buffer types seems ok to me. What I would however like is a unification of the two dynamic_buffer concepts, which are currently a mess. Taking the beast buffers and making them work nicely with asio would be great. But given the v1 v2 chaos, this might not be very easy. I would add that I think every buffer is also sink & source, so they maybe should implement that interface as well. Additionally, I feel some need for a library managing raw memory. It's quite common to just use vector<unsigned char>, but I find this incorrect, because it's typed. Raw memory doesn't have a type, which is why malloc returns void* and not unsigned char*. Thus a container type for a chunk of raw memory should also be included. I.e. something akin to a vector<void> if you will. Or like `bytes` in python. On Sat, Jan 28, 2023 at 3:35 AM Vinnie Falco via Boost < boost@lists.boost.org> wrote:
Greetings. I am asking if there is any interest in a Boost.Buffers library, which consists of the following:
* ConstBuffer, MutableBuffer concepts * ConstBuffers, MutableBuffers concepts * const_buffer, mutable_buffer types * DynamicBuffer concept * `source` and `sink` abstract interfaces used to define the Source and Sink concepts * metafunctions for checking all type requirements * implementations of common buffers and dynamic buffers: - buffers_pair - array_of_buffers - circular_buffer - dynamic_string_buffer * Algorithms: - buffer_copy - buffer_size
The twist is that this library has no dependency on Asio or Boost.Asio (but Asio will still recognize them thanks to some template heroics and a splash of forward declarations). Why am I proposing this? Well, let me tell you some things...
I am currently working on these two libraries: Boost.Http.Proto Boost.Http.Io
The Proto library implements HTTP/1 to a fuller feature set than Beast and fixes all the design flaws in Beast (but it is completely not API compatible). This library does not do any network I/O, it is expressed purely in terms of buffer concepts (i.e. depends on Boost.Buffer and not Boost.Asio).
The Io library depends on Asio and the Proto library to provide the network algorithms which use Asio to implement HTTP/1. This is the equivalent HTTP functionality of what Beast offers today.
Websockets is not included but when I am done with Http I will redo websockets with improvements and new features, and similarly they will be developed as two libraries:
Boost.Websockets.Proto Boost.Websockets.Io
My motivation for splitting libraries this way is as follows:
* Each library is smaller and compiles more quickly * CI turnarounds are faster * Separation of concerns
The HTTP Protocol library is designed in a way that much of an application's HTTP business logic can be expressed independently of Asio using only the constructs found in Http.Proto. This especially means fewer templates and faster compilation because Http.Proto is designed from the ground up to avoid templates whenever possible.
However during the development of Http.Proto I find myself having to reach for the same buffer algorithms and containers that I developed for Beast (such as the circular buffer and the dynamic buffer). So Http.Proto has been steadily growing these things:
< https://github.com/CPPAlliance/http_proto/blob/a3a284f06597ad12ca6cc1782bebb...
Extract this independent buffer functionality into a separate library Boost.Buffers would bring these benefits:
* Again a smaller library, faster compilation * Faster CI turnarounds * Separation of concerns
Additionally, algorithms can be expressed in terms of asio-independent buffer types and concepts. For example, Boost.JSON could implement the boost::buffers::sink and boost::buffers::source interfaces which would enable its use in Boost.Http.Proto and Boost.Requests (an upcoming HTTP client library from Klemens), without having to depend on either of those libraries or Asio.
Is this style of development of smaller, fine grained libraries that separate concerns something that is desirable for Boost? Glad for any feedback.
-- Regards, Vinnie
Follow me on GitHub: https://github.com/vinniefalco
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Klemens Morgenstern wrote:
Additionally, I feel some need for a library managing raw memory. It's quite common to just use vector<unsigned char>, but I find this incorrect, because it's typed. Raw memory doesn't have a type, which is why malloc returns void* and not unsigned char*.
Raw memory bytes do have a type, and it's `unsigned char`.
Thus a container type for a chunk of raw memory should also be included. I.e. something akin to a vector<void> if you will. Or like `bytes` in python.
You can in principle use std::byte for this, which is why it was added. Of course they had to ruin it by adding operations to it.
Klemens Morgenstern wrote:
Additionally, I feel some need for a library managing raw memory. It's quite common to just use vector<unsigned char>, but I find this incorrect, because it's typed. Raw memory doesn't have a type, which is why malloc returns void* and not unsigned char*.
Raw memory bytes do have a type, and it's `unsigned char`.
Then why do malloc and operator new return void* ?
Klemens Morgenstern wrote:
Klemens Morgenstern wrote:
Additionally, I feel some need for a library managing raw memory. It's quite common to just use vector<unsigned char>, but I find this incorrect, because it's typed. Raw memory doesn't have a type, which is why malloc returns void* and not unsigned char*.
Raw memory bytes do have a type, and it's `unsigned char`.
Then why do malloc and operator new return void* ?
Because in C, you can assign void* to X* without a cast.
On Fri, Jan 27, 2023 at 8:09 PM Klemens Morgenstern via Boost
wrote: I do like reducing dependencies, but I don't see the point of separating things just for the sake of it.
There are benefits to a strategic refactoring of composite libraries ("separating things") but that is not why I am proposing this. The answers to questions follow but before that I want to provide readers with a key insight gleaned from field experience. There is a recurring buffer motif which is a dual, that is: 1. reading from or writing into given buffers 2. providing buffers to be read from or written into These operations are fundamental, and one cannot be efficiently expressed in terms of the other. The two forms of this pattern occur with such frequency across varying domains that there is value in designing a common interface to express them. More on this later.
If you're writing a boost library most users will have installed boost as a monolith; but even those that didn't will use this in the context of http+asio so that an avoiding the inclusion of `asio/buffer.hpp` doesn't really seem like a great motivation for a new library to me.
We have libraries right now that implement serialization and parsing
for things such as JSON, HTTP message bodies, deflate or gzip
compression and decompression. Soon we will likely have Boost.Mustache
which applies parameters to a template to produce output. And having a
MIME/forms library is probably inevitable. We want to allow users to
use these things as payloads for incoming or outgoing HTTP requests
and responses.
Currently this is done by adding the support to the library which
implements the HTTP, such as Boost.Requests. Note how Boost.Requests
has to know about JSON and how to serialize or deserialize it:
https://github.com/CPPAlliance/requests/blob/1214608d19f6a8c020671fcd3507e00...
To add support for some new body X, the structural model in use by
Requests as written requires changing the Requests library itself,
because the interface to serialize or parse between X and HTTP
needlessly conflates networking and asio buffers as can be seen here:
https://github.com/CPPAlliance/requests/blob/1214608d19f6a8c020671fcd3507e00...
The maintainer of Requests has to manually add a new asynchronous
algorithm for every HTTP method in order to receive X as a body type.
Here are the signatures for receiving a response containing body X for
the HTTP methods GET, POST, and PUT. There are more methods which need
to be supported but I have not bothered to link them, check the file
to see them:
https://github.com/CPPAlliance/requests/blob/1214608d19f6a8c020671fcd3507e00...
<https://github.com/CPPAlliance/requests/blob/1214608d19f6a8c020671fcd3507e00...
https://github.com/CPPAlliance/requests/blob/1214608d19f6a8c020671fcd3507e00...
Every time Requests wants to support a new body type, it must add as a
dependency the library containing the type it wishes to support. And
then it must re-implement that support four times for every HTTP
method: two synchronous versions (with and without exceptions) and two
asynchronous versions (with and without exceptions). If
Boost.Http.Proto and Beast used this model, they too would have to
depend on every library which has a candidate body, and duplicate the
serialization and deserialization algorithms (called BodyReader and
BodyWriter in Beast).
This is clearly unsustainable.
I propose a better solution which eliminates almost all of the
overloads by refactoring to separate concerns and allows the
algorithms to be expressed once without introducing multiple
dependencies.
Instead of teaching Requests how to serialize and deserialize every X
it wants to support, Boost.Buffers provides asio-agnostic concepts
which allow the library containing X itself to provide the
serialization and deserialization algorithms. Boost.JSON might add
this declaration allowing a json::value to be serialized using
Boost.Buffers:
#include
I for one don't mind using sub-libraries, e.g. including boost/core for the buffer types seems ok to me.
Having asio-agnostic buffers in core is certainly better than not having them at all, but having them in their own library is better still. It gets its own library-specific documentation. It runs CI jobs faster and consumes less overall Drone resources over time. It becomes more discoverable by users. We can follow the principle of one type or concern per header file instead of cramming everything into a single buffer.hpp file. This lets downstream libraries decide which headers they actually need, to cut down on compile times. We can add useful buffer and dynamic buffer implementations such as buffers_pair, circular_buffer, flat_buffer, and the perennial favorite array_of_buffers (supporting sizes of up to one million or more). Additionally the library can be evolved independently of the Asio author which in simple terms means that open issues will receive immediate attention, emails will get answers, and the maintainers can be reached on Slack. As you said, "most users will have installed boost as a monolith." This means there is no penalty for its existence as its own Boost library. But there are the benefits stated above. These benefits are not theoretical; Requests aims to add forms, files, and various other things. When these are expressed in terms of Boost.Buffers, they will automatically work in Requests without the need to add support for it specifically. But there is another huge benefit, every library which adds support for parsing and/or serialization in terms of Boost.Buffer concepts can now be used in Boost.Http.Proto with no modification or changing of library dependencies! At this point, someone like Peter might object. He might say, well Boost.JSON doesn't care about sources and sinks it only cares about JSON. Or worse he might become dismissive if I ask him to support Boost.Buffers in the upcoming Boost.Mustache. But what is the purpose of having this giant monolith of libraries, if not to take advantage of our ability to make all the libraries work together seamlessly with the other libraries, and enhance the value of the collection as a whole?
What I would however like is a unification of the two dynamic_buffer concepts, which are currently a mess. Taking the beast buffers and making them work nicely with asio would be great.
This is a good goal, but it is orthogonal to what I am proposing here. Boost.Buffers does not use Asio, does not depend on Asio, does not require its downstream libraries to use Asio, and does not replace Asio for libraries that need to perform networking. The fact that it uses concepts that are astoundingly similar to Asio merely demonstrates that the Asio author discovered very good general purpose abstractions.
Thus a container type for a chunk of raw memory should also be included. I.e. something akin to a vector<void> if you will.
Hmm... I do not believe std::vector<void> can be made to work but we can certainly do std::vector<unsigned char> (and we should). Thanks
Greetings. I am asking if there is any interest in a Boost.Buffers library, which consists of the following:
I like the separation of concerns, so I would endorse this. I would use this in two ways in MySQL: * As an implementation detail for internal buffers. I would probably be alright with something like Python's bytes (as Klements proposed). * In the interface, I need types to represent "an owning blob" and "an owning blob view". I don't know if this library would cover this, but would be nice to see it. Regards, Ruben.
On 28/01/2023 08:34, Vinnie Falco wrote:
Greetings. I am asking if there is any interest in a Boost.Buffers library, which consists of the following:
* ConstBuffer, MutableBuffer concepts * ConstBuffers, MutableBuffers concepts * const_buffer, mutable_buffer types * DynamicBuffer concept * `source` and `sink` abstract interfaces used to define the Source and Sink concepts * metafunctions for checking all type requirements * implementations of common buffers and dynamic buffers: - buffers_pair - array_of_buffers - circular_buffer - dynamic_string_buffer * Algorithms: - buffer_copy - buffer_size
I assume also adapters from std::span<T> and ranges thereof, since those are the New Hotness? I am a little hesitant in breaking this off from Asio, since as Klemens pointed out it would be unusual to not have the header available, and since whatever is in Asio is already somewhat-standards-adjacent with some third party implementations already, making them _almost_ vocabulary types anyway. (Though of course "almost" compiles no code.) (I don't know whether that makes it better or worse if your types are identical to the ASIO ones or "improved" in some way.)
Le 2023-01-31 07:30, Gavin Lambert via Boost a écrit :
On 28/01/2023 08:34, Vinnie Falco wrote:
Greetings. I am asking if there is any interest in a Boost.Buffers library, which consists of the following:
I am a little hesitant in breaking this off from Asio, since as Klemens pointed out it would be unusual to not have the header available, and since whatever is in Asio is already somewhat-standards-adjacent with some third party implementations already, making them _almost_ vocabulary types anyway. (Though of course "almost" compiles no code.)
A buffer concept has use cases outside the usual targets of asio (think of MCUs). C++/Boost currently lack a fixed size circular_buffer implementation, for example. That being said, i think we currently have a vocabulary type for non-owning non-resizable buffers: it's called span<byte>. Maybe an owning span would make some sense, to be able to transfer ownership of the underlying memory. Outside this, i'm not sure. IMHO a buffer is often just a special use case of a container. Regards, Julien
On Tue, Jan 31, 2023 at 2:08 AM Julien Blanc via Boost
That being said, i think we currently have a vocabulary type for non-owning non-resizable buffers: it's called span<byte>.
span<byte> is problematic for a few reasons. There's nothing wrong if users want to use this type to describe a buffer, but it is less than ideal as a vocabulary type. Regardless, in order to make Boost.Buffers types and concepts work seamlessly with Asio (but without actually depending on Asio) it is necessary for the two fundamental buffer types `const_buffer` and `mutable_buffer` to be declared by the library rather than being a span of things.
Maybe an owning span would make some sense, to be able to transfer ownership of the underlying memory.
Beast innovates here with its own custom containers but I think I will just use std::string and std::vector as they do the same thing and require less work.
Outside this, i'm not sure. IMHO a buffer is often just a special use case of a container.
Right, dynamic buffer adaptors control a string or vector. Regards
On Mon, Jan 30, 2023 at 10:31 PM Gavin Lambert via Boost
I assume also adapters from std::span<T> and ranges thereof, since those are the New Hotness?
Adapters yes but I don't know what the ranges is (probably yes to that if it makes sense).
I am a little hesitant in breaking this off from Asio, since as Klemens pointed out it would be unusual to not have the header available,
I wouldn't want Boost.JSON to depend on Boost.Asio. But I am OK with Boost.JSON depending on Boost.Buffers.
...since whatever is in Asio is already somewhat-standards-adjacent
Well the Networking TS was rejected.
(I don't know whether that makes it better or worse if your types are identical to the ASIO ones or "improved" in some way.)
I'm trying to keep things as similar as possible if for no other reason, that introducing differences can only lead to confusion. However I am fixing what I perceive to be some design flaws; wg21 has a negative impact on Asio's development. Thanks
pt., 27 sty 2023 o 20:35 Vinnie Falco via Boost
Greetings. I am asking if there is any interest in a Boost.Buffers library, which consists of the following:
* ConstBuffer, MutableBuffer concepts * ConstBuffers, MutableBuffers concepts * const_buffer, mutable_buffer types * DynamicBuffer concept * `source` and `sink` abstract interfaces used to define the Source and Sink concepts * metafunctions for checking all type requirements * implementations of common buffers and dynamic buffers: - buffers_pair - array_of_buffers - circular_buffer - dynamic_string_buffer * Algorithms: - buffer_copy - buffer_size
The twist is that this library has no dependency on Asio or Boost.Asio (but Asio will still recognize them thanks to some template heroics and a splash of forward declarations). Why am I proposing this? Well, let me tell you some things...
I am currently working on these two libraries: Boost.Http.Proto Boost.Http.Io
The Proto library implements HTTP/1 to a fuller feature set than Beast and fixes all the design flaws in Beast (but it is completely not API compatible). This library does not do any network I/O, it is expressed purely in terms of buffer concepts (i.e. depends on Boost.Buffer and not Boost.Asio).
The Io library depends on Asio and the Proto library to provide the network algorithms which use Asio to implement HTTP/1. This is the equivalent HTTP functionality of what Beast offers today.
Websockets is not included but when I am done with Http I will redo websockets with improvements and new features, and similarly they will be developed as two libraries:
Boost.Websockets.Proto Boost.Websockets.Io
My motivation for splitting libraries this way is as follows:
* Each library is smaller and compiles more quickly * CI turnarounds are faster * Separation of concerns
The HTTP Protocol library is designed in a way that much of an application's HTTP business logic can be expressed independently of Asio using only the constructs found in Http.Proto. This especially means fewer templates and faster compilation because Http.Proto is designed from the ground up to avoid templates whenever possible.
However during the development of Http.Proto I find myself having to reach for the same buffer algorithms and containers that I developed for Beast (such as the circular buffer and the dynamic buffer). So Http.Proto has been steadily growing these things:
< https://github.com/CPPAlliance/http_proto/blob/a3a284f06597ad12ca6cc1782bebb...
Extract this independent buffer functionality into a separate library Boost.Buffers would bring these benefits:
* Again a smaller library, faster compilation * Faster CI turnarounds * Separation of concerns
Additionally, algorithms can be expressed in terms of asio-independent buffer types and concepts. For example, Boost.JSON could implement the boost::buffers::sink and boost::buffers::source interfaces which would enable its use in Boost.Http.Proto and Boost.Requests (an upcoming HTTP client library from Klemens), without having to depend on either of those libraries or Asio.
Is this style of development of smaller, fine grained libraries that separate concerns something that is desirable for Boost? Glad for any feedback.
This is an idea certainly worth exploring. Let me share some random observations. The idea of vocabulary types is great and I would like Boost to be the place for hosting them. In the past, vocabulary types were `optional`, `tuple`, `variant`, `shared_ptr`. Today the basics for networking seems like good candidates for vocabulary types. Is Boost.Buffer a candidate for such a vocabulary type? First question, is this library to be consumed directly by user programs, or only by other libraries, such as Boost.Beast and Boost.Http.Proto? Is it possible that the final program that uses Boost.Buffer will not use Boost.ASIO? For me, as a consumer of networking libraries, who can afford to download and build the entire set of Boost libraries, the biggest advantage would be that Buffers would be well documented and tested. I am not convinced by the argument with Boost.JSON not depending on ASIO. We have (I think a still unresolved issue) of Boost-serializing Boost.Optional. Should the code responsible for the serialization of Boost.Optional be part of Boost.Serialization or Boost.Optional? But we will certainly not solve it by introducing yet another library. There are other ways to solve it, say, through optional headers. Also, Boost.JSON doesn't need to depend on the implementation of the circular buffer. But here I am only criticizing the argument with Boost.JSON, not the idea to factor out Boost.Buffers. Regards, &rzej;
On 3/02/2023 10:57, Andrzej Krzemienski wrote:
We have (I think a still unresolved issue) of Boost-serializing Boost.Optional. Should the code responsible for the serialization of Boost.Optional be part of Boost.Serialization or Boost.Optional? But we will certainly not solve it by introducing yet another library. There are other ways to solve it, say, through optional headers.
I think the two schools of thought on that are: 1. Serialization came first and so Optional should take it into account in its implementation. 2. Optional is more fundamental than Serialization, and you're more likely to use optionals than serialization, so the latter should have the supporting code. I personally find the second argument more compelling (although the waters are muddied by there now being a std::optional too, increasing the chance you might want serialization without caring about boost::optional), but that only solves the problem of which repository the bulk of code should belong to. Unfortunately in C++ there's no way to say "if these two otherwise unrelated headers were included then also include this", short of order-sensitive preprocessor checks. This then I think leads to only three reasonable solutions: 1. The user has to know that the integration exists and explicitly #include an additional header file (e.g. boost/serialization/optional.hpp) when they're using both together. 2. *Both* libraries need to detect the other and include the integration header from their top-level wrapper regardless of the original include order. So if the user happens to include both libraries "in general" then they get the integration automatically. 3. One library decides that the other is sufficiently "vocabulary" and/or small enough to always be loaded if the first is, and just unconditionally includes it (from the perspective of the "whole library" header, at least). For this particular case, Boost appears to have chosen #1 currently, and doesn't implement #2 or #3. These are not mutually exclusive. #2 is the most user-friendly option (#1 is less discoverable), but it's not readily extensible for detecting if std types are included, or for third-party library types unaware of Boost, making it rather awkward in practice for anything non-Boost. I'm not entirely sure how this all applies to the Buffers/Asio situation. Logically, Buffers would be more fundamental than Asio, so it would make sense for Asio to be modified to use/be compatible with Buffers (rather than the reverse, as currently seems to be proposed), though I'm not sure how well that would catch on since Asio will likely want to retain its own buffer implementations for standalone use at least.
On Thu, Feb 2, 2023 at 2:47 PM Gavin Lambert via Boost
I'm not entirely sure how this all applies to the Buffers/Asio situation. Logically, Buffers would be more fundamental than Asio, so it would make sense for Asio to be modified to use/be compatible with Buffers (rather than the reverse, as currently seems to be proposed), though I'm not sure how well that would catch on since Asio will likely want to retain its own buffer implementations for standalone use at least.
Well as I mentioned in my previous message, I think that serialization is a different thing from where Buffers would be applicable. As an interesting note. Boost.JSON handles conversion of OptionalLike types to JSON when performing json::value_from and when serializing (soon). We did this not by depending on Boost.Optional but rather, defining a concept for all OptionalLike types. Generally speaking I think if a library wants to support types that match certain generic concepts, then the place for that support goes into the library itself. I would be in favor of a Boost library which offered metafunctions for identifying types which match certain models such as OptionalLike, MapLike, StringLike (e.g convertibility to string_view), TupleLike, VariantLike and such. JSON could use that and perhaps Serialization. It might be worth exploring. I have designed Boost.Buffers to work with Asio (but without requiring Asio) to make it palatable to use. Asio changing is extremely unlikely so the burden is on Buffers to be interoperable. On a separate note all of this talk about headers only working when other headers are "present" or anything like that, makes me uncomfortable. I like it when a build always produces one version of the library no matter what else is installed or what macros are set. I've tried to get fancy with includes and I always regretted it. Thanks
pt., 3 lut 2023 o 02:13 Vinnie Falco via Boost
On Thu, Feb 2, 2023 at 2:47 PM Gavin Lambert via Boost
wrote: I'm not entirely sure how this all applies to the Buffers/Asio situation. Logically, Buffers would be more fundamental than Asio, so it would make sense for Asio to be modified to use/be compatible with Buffers (rather than the reverse, as currently seems to be proposed), though I'm not sure how well that would catch on since Asio will likely want to retain its own buffer implementations for standalone use at least.
Well as I mentioned in my previous message, I think that serialization is a different thing from where Buffers would be applicable. As an interesting note. Boost.JSON handles conversion of OptionalLike types to JSON when performing json::value_from and when serializing (soon). We did this not by depending on Boost.Optional but rather, defining a concept for all OptionalLike types. Generally speaking I think if a library wants to support types that match certain generic concepts, then the place for that support goes into the library itself.
On the specific topic of OptionalLike, note that we have i Boost the concept of OptionalPointee: https://github.com/boostorg/utility/blob/c960bef6ef005413d96cc7366b66cbdf56d... It is not well advertised though. It was added along Boost.Optional, but it was not clear where to put it.
I would be in favor of a Boost library which offered metafunctions for identifying types which match certain models such as OptionalLike, MapLike, StringLike (e.g convertibility to string_view), TupleLike, VariantLike and such. JSON could use that and perhaps Serialization. It might be worth exploring.
Agreed. A library for enforcing concepts from Boost.Libraries. Regards, &rzej;
I have designed Boost.Buffers to work with Asio (but without requiring Asio) to make it palatable to use. Asio changing is extremely unlikely so the burden is on Buffers to be interoperable.
On a separate note all of this talk about headers only working when other headers are "present" or anything like that, makes me uncomfortable. I like it when a build always produces one version of the library no matter what else is installed or what macros are set. I've tried to get fancy with includes and I always regretted it.
Thanks
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 2/2/23 2:46 PM, Gavin Lambert via Boost wrote:
On 3/02/2023 10:57, Andrzej Krzemienski wrote:
We have (I think a still unresolved issue) of Boost-serializing Boost.Optional. Should the code responsible for the serialization of Boost.Optional be part of Boost.Serialization or Boost.Optional? But we will certainly not solve it by introducing yet another library. There are other ways to solve it, say, through optional headers.
I think the two schools of thought on that are:
1. Serialization came first and so Optional should take it into account in its implementation. 2. Optional is more fundamental than Serialization, and you're more likely to use optionals than serialization, so the latter should have the supporting code.
I personally find the second argument more compelling (although the waters are muddied by there now being a std::optional too, increasing the chance you might want serialization without caring about boost::optional), but that only solves the problem of which repository the bulk of code should belong to.
Unfortunately in C++ there's no way to say "if these two otherwise unrelated headers were included then also include this", short of order-sensitive preprocessor checks. This then I think leads to only three reasonable solutions:
1. The user has to know that the integration exists and explicitly #include an additional header file (e.g. boost/serialization/optional.hpp) when they're using both together.
or ... boost/optional/serialization.hpp ?
2. *Both* libraries need to detect the other and include the integration header from their top-level wrapper regardless of the original include order. So if the user happens to include both libraries "in general" then they get the integration automatically.
3. One library decides that the other is sufficiently "vocabulary" and/or small enough to always be loaded if the first is, and just unconditionally includes it (from the perspective of the "whole library" header, at least).
For this particular case, Boost appears to have chosen #1 currently, and doesn't implement #2 or #3. These are not mutually exclusive.
I have chose the following rule. a) if it's in the standard, it's in boost/optional b) if it's somewhere else it's in boost/somewhere_else/optional.hpp Adding pieces of every other library into boost serialization doesn't scale and is not maintainable - at least no by me. If a user includes boost/optional/serialization.hpp "too high" in the inclusion tree it creates spurious dependencies - not a great thing. If one includes a test for serialization inside a library, it makes the whole library dependent on the serialization library which in turn ends up depending on large parts of boost. Not what a lot of people want. In twenty years we've never been able to resolve this. I believe there are few other "utility types" which have this same problem.
#2 is the most user-friendly option (#1 is less discoverable), but it's not readily extensible for detecting if std types are included, or for third-party library types unaware of Boost, making it rather awkward in practice for anything non-Boost.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 4/02/2023 07:14, Robert Ramey wrote:
1. The user has to know that the integration exists and explicitly #include an additional header file (e.g. boost/serialization/optional.hpp) when they're using both together.
or ... boost/optional/serialization.hpp ?
Perhaps, though as I said I find the "optional is more fundamental" argument more compelling, so by that logic the compatibility code should be in serialization and not optional. Also, boost/serialization/optional.hpp already does exist, and boost/optional/serialization.hpp does not. Though having both header files exist (with one merely a redirect to the canonical one) could be an interesting choice, to allow the user to not have to think about which one is more fundamental. (Though if one file exists and one does not, that becomes obvious fairly quickly.) One downside of the current serialization implementation is that (as far as I can see) it *only* supports boost::optional and not std::optional, and the "obvious" header name (boost/serialization/optional.hpp) is already taken. Were it to extend support for std::optional in the future, how should that look? Just detect C++17 and then assume you want it automatically? Try to auto-detect a previous #include <optional>? Assume including boost/serialization/optional.hpp means you want both? Introduce boost/serialization/std_optional.hpp? All of these choices have pros and cons. Another variant might be mixing several of these, by renaming current optional.hpp to boost_optional.hpp, adding std_optional.hpp, and then a new optional.hpp that includes one, the other, or both based on C++17 support (or detecting prior include of Boost.Optional), though there are several caveats there too.
On Mon, Feb 6, 2023 at 2:43 PM Gavin Lambert via Boost
Perhaps, though as I said I find the "optional is more fundamental" argument more compelling, so by that logic the compatibility code should be in serialization and not optional.
I just want to underscore that the question of "where does optional go" is completely orthogonal to this proposed Boost.Buffers library, and the buffers::source and buffers::sink interfaces in particular. My proposed library has nothing to do with serialization and everything to do with providing buffer-oriented interfaces. In fact, buffers::source is more fundamental than serialization. Boost.Serialization sits between buffers::source and the algorithm for serialization of an optional. This diagram shows the relationship: { host } <---> Boost.Buffers <---> Boost.Serialization <---> { optional } In this diagram "host" is the host program or library which is implemented in terms of Boost.Buffer's abstract interfaces (buffers::source for the case of using Boost.Serialization). As seen in the diagram, the question of where the serialization spec for "optional" goes, is unrelated to Boost.Buffers. Another way to think of it is like this. If Boost.Serialization implements the buffers::source interface, then any host which uses Boost.Buffers will be able to make use of types archived through Boost.Serialization. I'm not saying whether this is particularly useful or not as that would depend on the specific use-case. For the upcoming Boost.Http.Proto library, this means that anything which defines a serialization for Boost.Serialization could be attached as the body of an HTTP message. The diagram for Boost.JSON looks different: { host } <---> Boost.Buffers <---> Boost.JSON Here, if Boost.JSON wraps its existing streaming serializer for the buffers::source interface then any Boost.Buffers-enabled host can make use of JSON sources without depending on the Boost.JSON library. There's a network effect at play; every library that enables Boost.Buffers geometrically increases the utility of every other library that uses Boost.Buffers. Furthermore, this scales to any number of libraries and resolves the question of where the buffers::sink (or buffers::source) algorithm goes. It goes in the library which already implements a buffer-oriented data type. This includes but is not limited to: * JSON * XML * Boost.Serialization archives * Mustache templates and output Thanks
wt., 7 lut 2023 o 00:31 Vinnie Falco via Boost
On Mon, Feb 6, 2023 at 2:43 PM Gavin Lambert via Boost
wrote: Perhaps, though as I said I find the "optional is more fundamental" argument more compelling, so by that logic the compatibility code should be in serialization and not optional.
I just want to underscore that the question of "where does optional go" is completely orthogonal to this proposed Boost.Buffers library, and the buffers::source and buffers::sink interfaces in particular. My proposed library has nothing to do with serialization and everything to do with providing buffer-oriented interfaces.
In fact, buffers::source is more fundamental than serialization. Boost.Serialization sits between buffers::source and the algorithm for serialization of an optional. This diagram shows the relationship:
{ host } <---> Boost.Buffers <---> Boost.Serialization <---> { optional }
In this diagram "host" is the host program or library which is implemented in terms of Boost.Buffer's abstract interfaces (buffers::source for the case of using Boost.Serialization). As seen in the diagram, the question of where the serialization spec for "optional" goes, is unrelated to Boost.Buffers.
Another way to think of it is like this. If Boost.Serialization implements the buffers::source interface, then any host which uses Boost.Buffers will be able to make use of types archived through Boost.Serialization. I'm not saying whether this is particularly useful or not as that would depend on the specific use-case. For the upcoming Boost.Http.Proto library, this means that anything which defines a serialization for Boost.Serialization could be attached as the body of an HTTP message.
The diagram for Boost.JSON looks different:
{ host } <---> Boost.Buffers <---> Boost.JSON
Here, if Boost.JSON wraps its existing streaming serializer for the buffers::source interface then any Boost.Buffers-enabled host can make use of JSON sources without depending on the Boost.JSON library. There's a network effect at play; every library that enables Boost.Buffers geometrically increases the utility of every other library that uses Boost.Buffers. Furthermore, this scales to any number of libraries and resolves the question of where the buffers::sink (or buffers::source) algorithm goes. It goes in the library which already implements a buffer-oriented data type. This includes but is not limited to:
* JSON * XML * Boost.Serialization archives * Mustache templates and output
As indicated earlier, I support the extraction of Boost.Buffers into a separate Boost library. Let me offer some further thoughts. 1. It is more of a question. These buffers::sink and buffers::source being mentioned in this thread: are they already present in one of the Boost libraries (ASIO, Beast)? 2. For any "vocabulary interface" we could separate out three parts: (1) the interface, (2) the sinks, (3) the sources. I am slightly abusing the terms, so let me give an example of the STL. Iterators are the interface. The algorithms are "sinks": they do not have to know about containers. The containers are the "sources" they do not have to know about iterators. If the STL was a Boost library, the only thing that other Boost libraries would depend on would be iterators -- not the algorithms, not the containers -- because it is only the iterators -- the interface -- that lets you plug your library into the system. I see the same situation with the proposed Boost.Buffers. The only thing that multiple libraries (Boost and non-Boost) would benefit from is the buffer interface (boost::sink, boost::source). No buffer implementations, no buffer algorithms. As you mentioned earlier, things like a circular buffer are shared between at least two libraries, so there is a merit in separating this, but this is just an engineering trade-off, rather than a vocabulary interface design (like boost::sink, boost::source). This remark applies to every library with a sound vocabulary-like interface: the interface should go separately. But that would be a very small library, maybe not worth a separate infrastructure. 3. There are a number of interfaces, where everyone can plug their type, in the STD and Boost that have an overlap. We have the IOStream interface, we have Boost.Serialization interface, std::format is coming, and we have now Boost.Buffer being proposed. Can you make a clear distinction why Boost.Buffers is different? Why do we need another one? Are previous ones defective (and can be superseded), or do they play a different, incompatible role? I suppose that buffers have much to do with buffering, that is working with chunks of messages. But how does this work with JSON? Can you even start thinking about parsing JSON if you only have a part, broken at an arbitrary position? Also, IOStreams have a layer of buffers. What is the relation there (Between IOStream buffers and the proposed Boost.Buffers)? 4. Would Boost.Buffers satisfy my every use case for buffers? Are buffers only about how you allocate a new chunk of memory? Or are they about identifing a place where you can cut your message into meaningful portions? Regards, &rzej;
On Mon, Feb 6, 2023 at 11:40 PM Andrzej Krzemienski
Let me offer some further thoughts.
Yes these are very good :)
1. It is more of a question. These buffers::sink and buffers::source being mentioned in this thread: are they already present in one of the Boost libraries (ASIO, Beast)?
These are new. The motivation for these types is originally for type-erasing certain HTTP bodies during parsing or serialization in the new Http.Proto library (https://github.com/CPPAlliance/http_proto). This library implements HTTP/1 and makes some different design choices which aim to fix some inherent defects in Beast's design.
The only thing that multiple libraries (Boost and non-Boost) would benefit from is the buffer interface (boost::sink, boost::source). No buffer implementations, no buffer algorithms.
The situation in Beast is that it is actually several libraries in one: 1. "sans-IO" HTTP/1 2. HTTP/1 on Asio 3. "sans-IO" Websocket 4. Websocket on Asio 5. Buffer algorithms and containers 6. Asio utilities 7. C++ port of ZLib (!) The "core" directory contain the files for items 5 and 6 above: https://github.com/boostorg/beast/tree/341ac7591b2b023c81de13312a80d1e824742... In theory there is nothing wrong with aggregating these things into one library. But for practical reasons, quite frankly it sucks. It takes forever for CI to turn around, the docs end up taking longer to build because there is so much more stuff, and it just gives off a bulky vibe that isn't fun to work with. One of my goals for my new generation of libraries (which intend to replace Beast) is to design things in a way that they do more with less API and implementation. Less "try-hard" so to speak. In Http.Proto I tried very hard to stay away from having to need these various buffer implementations and concepts but in the end it proved unworkable. It turns out that these buffer algorithms and containers are just so damn useful that even in a "sans-IO" (https://sans-io.readthedocs.io/) library they end up being the correct choice. Okay fine so I brought back in some selected buffer related things to Http.Proto but only as implementation details and private interfaces. No problem the library stays lean. But... oh well that didn't work out so well either because as it turns out, using buffer sequences to define the HTTP body is very useful and that brought me right back to where I started which is that the HTTP protocol library API benefits from buffer concepts. And the user benefits from having implementations of buffers on hand. Specifically, HTTP/1 message body serialization and parsing should support: Three body styles for `serializer` 1 Specify a ConstBufferSequence 2 Specify a Source 3 Write into a serializer::stream Three body styles for `parser` 4 Specify a DynamicBuffer 5 Specify a Sink 6 Read from a parser::stream Boost.Buffers fulfills the API requirements for achieving 1,2,4, and 5 above.
3. There are a number of interfaces, where everyone can plug their type, in the STD and Boost that have an overlap. We have the IOStream interface, we have Boost.Serialization interface, std::format is coming, and we have now Boost.Buffer being proposed. Can you make a clear distinction why Boost.Buffers is different? Why do we need another one? Are previous ones defective (and can be superseded), or do they play a different, incompatible role?
Yes. std::istream, std::ostream: These are actually pretty good substitutes for source and sink. They are in the standard already. They perform type-erasure. And they come with implementations (e.g. stringstream, ofstream). They could in theory work, and many types already support operator<< to std::ostream& so these could be generically used. Good thinking Andrzej :) But it is not without problems. It's got weird error handling and signaling for end of stream. Implementing your own istream or ostream can be difficult. It isn't design from the ground up for the buffer-oriented interface They are biased for character -based output and part of their interface has to do with formatting (which is out of scope for Boost.Buffers). Boost.Serialization: This is an entirely different thing from buffer-oriented exchange of data. Rather it is about defining algorithms for transporting types to and from an "archive" at full fidelity. This is out of scope. Boost.Buffers could have a say in how an archive represented as zero or more contiguous spans of bytes might be transported or streamed from one API to another. But it has nothing to say (nor cares) about how the user defined types map to or from those bytes. std::format: Kind of the same situation as Boost.Serialization. It defines a way to convert types into ASCII text and substitute that text into a larger corpus, which is out of scope. Boost.Buffers could have a say in how the buffers produced by std::format might be transported to another API.
I suppose that buffers have much to do with buffering, that is working with chunks of messages. But how does this work with JSON? Can you even start thinking about parsing JSON if you only have a part, broken at an arbitrary position?
As a matter of fact... you can :) Boost.JSON is unique in that it is the only JSON library which comes with a streaming parser and a streaming serializer, to allow buffer-at-a-time processing. This is essential for network programs to provide fairness. Specifically the streaming interface allows the implementor to restrict the amount of work performed when serializing or parsing JSON, and spread the computational requirements of handling large JSON texts across multiple I/O cycles so that one connection does not monopolize a thread. You can see those interfaces here: https://www.boost.org/doc/libs/1_81_0/libs/json/doc/html/json/input_output.h... https://www.boost.org/doc/libs/1_81_0/libs/json/doc/html/json/ref/boost__jso...
Also, IOStreams have a layer of buffers. What is the relation there (Between IOStream buffers and the proposed Boost.Buffers)?
I have to be honest I find the IOStreams interfaces incredibly confusing with the buffers and the "controlled sequence" and the g pointer and the p pointer and the.. well, you get the point. Maybe you could tell me what the relationship is, if any, as I am not quite sure...
4. Would Boost.Buffers satisfy my every use case for buffers?
Well I don't know. You'd have to list the use-cases :) My recipe for this library was to start with Asio concepts, add in some implementations which end up being needed often, and add my own buffer-oriented filter, source, and sink abstract interfaces. If these are insufficient for a particular use case I would need to study the use-case and figure out if it is the right fit for Buffers and how it could be satisfied.
Are buffers only about how you allocate a new chunk of memory?
No. There are roughly three areas of interest: 1. range-like sequences of contiguous storage: - ConstBufferSequence, MutableBufferSequence - buffers::const_buffer, buffers::mutable_buffer 2. stream-like controlled buffers: - DynamicBuffer - buffers::circular_buffer, buffers::flat_buffer, buffers::string_buffer 3. abstract buffering interfaces: buffers::filter, buffers::source, buffers:sink In 1 the ranges are static, buffers can't change size nor can the range change size. The concepts are akin to `span const`. In 2 the controlled buffers can grow (via prepare/commit) and shrink (via consume). Depending on the implementation this could allocate memory (or not). For 3 these interfaces define how buffers are passed from one program interface to another, when operating a buffer-at-a-time processing algorithm.
Or are they about identifying a place where you can cut your message into meaningful portions?
No, but if you have already cut your message into zero or more contiguous bytes of storage then Boost.Buffers can help you do things with it. Thanks
On 2/6/23 2:43 PM, Gavin Lambert via Boost wrote:
On 4/02/2023 07:14, Robert Ramey wrote:
1. The user has to know that the integration exists and explicitly #include an additional header file (e.g. boost/serialization/optional.hpp) when they're using both together.
or ... boost/optional/serialization.hpp ?
I think we're off-topic - but what the hell.
Perhaps, though as I said I find the "optional is more fundamental" argument more compelling, so by that logic the compatibility code should be in serialization and not optional.
My rule has always been: if it's in the standard library, put it in boost/serialization/... otherwise, put it in boost/<library name>/serialization.hpp I didn't see any natural place for the standard library serializations so I put them boost/serialization. A big problem is that a number of library maintainers object to putting the serialization code for their data types into their libraries and won't do it. This explains the "optional" anomaly you've noted. They don't like it because our "dependency checking tools" Will show a dependency on serialization if the serialization library is referred to anywhere in a library, even if only in tests! Basically "dependency" would depend on whether an application actually serializes that type. Of course if one is building a shared library it's different - everything has to be included. Sorry for the wordy explanation, but I didn't have time to make it shorter.
Also, boost/serialization/optional.hpp already does exist, and boost/optional/serialization.hpp does not.
Because the maintainer of optional has chosen not to. If I put it in serialization, I'm signing on to adding/maintaining serialization types in all libraries. Probably not a wise idea. I don't want the work and I'm 75 years old.
Though having both header files exist (with one merely a redirect to the canonical one) could be an interesting choice, to allow the user to not have to think about which one is more fundamental. (Though if one file exists and one does not, that becomes obvious fairly quickly.)
So it's not a question of which is more "fundamental", which would always be subjective, it's just the simple rule above.
One downside of the current serialization implementation is that (as far as I can see) it *only* supports boost::optional and not std::optional, and the "obvious" header name (boost/serialization/optional.hpp) is already taken. Were it to extend support for std::optional in the future, how should that look? Just detect C++17 and then assume you want it automatically? Try to auto-detect a previous #include <optional>? Assume including boost/serialization/optional.hpp means you want both? Introduce boost/serialization/std_optional.hpp? All of these choices have pros and cons.
I think there is an implementation of serialization posted as a PR.
Another variant might be mixing several of these, by renaming current optional.hpp to boost_optional.hpp, adding std_optional.hpp, and then a new optional.hpp that includes one, the other, or both based on C++17 support (or detecting prior include of Boost.Optional), though there are several caveats there too.
This really is a question regarding "bridge libraries" which have a foot in more than library. The "real" solution would be a separate "library" which would contain just the serialization implementations for all types.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Thu, Feb 2, 2023 at 1:58 PM Andrzej Krzemienski
Today the basics for networking seems like good candidates for vocabulary types. Is Boost.Buffer a candidate for such a vocabulary type?
In theory, yes. The intent is for Buffers to be more general than just networking.
First question, is this library to be consumed directly by user programs, or only by other libraries, such as Boost.Beast and Boost.Http.Proto?
User programs could use it when they want to express algorithms in terms of ranges of buffers. My personal use-case is Beast, HTTP, Websocket, and JSON but this could be extended to say, base64 encoding and decoding, calculating SHA-2 digest over a range of buffers, simplified front end to things like zlib. Example: template< class DynamicBufferOut, class DynamicBufferIn > result< void > deflate( z_params&, DynamicBufferOut& DynamicBufferIn&, bool end_of_data);
Is it possible that the final program that uses Boost.Buffer will not use Boost.ASIO?
That is possible, yes.
For me, as a consumer of networking libraries, who can afford to download and build the entire set of Boost libraries, the biggest advantage would be that Buffers would be well documented and tested.
That is an advantage yes but in my opinion the advantage is that you get access to a rich and useful set of algorithms, containers, and concepts, without also depending on a particular network implementation.
I am not convinced by the argument with Boost.JSON not depending on ASIO.
Well, users definitely do not want to drag in Asio with their JSON.
We have (I think a still unresolved issue) of Boost-serializing Boost.Optional. Should the code responsible for the serialization of Boost.Optional be part of Boost.Serialization or Boost.Optional? But we will certainly not solve it by introducing yet another library.
I think that the cases of Boost.Serialize with boost::optional, and serialization of JSON using Boost.JSON and Boost.Buffers are two completely different cases. Boost.Serialize does not define a serialization format but rather a framework for allowing the user to define serialization. Boost.JSON is something else entirely; it is a library which implements a specific protocol for describing data using structured text. If you look at Boost.JSON you see that it has json::parser and json::serializer. These classes work a buffer at a time and they support streaming; that is, that the entire input or output need not be presented at once in order to make progress. How does Boost.Buffers help here? Well.. it provides the means for a library such as JSON, which has algorithms to produce or consume a buffer at a time, to type-erase its algorithm using the buffers::source or buffers::sink interfaces: /// An algorithm which consumes filled buffers struct sink { virtual ~sink() = 0; /** Called when data is available */ virtual result<void> write( span< const_buffer const > ) = 0; }; /// An algorithm which produces filled buffers struct source { /** The results of reading from the source. */ struct results { error_code ec; std::size_t bytes = 0; bool more = false; }; virtual ~source() = 0; /** Called when more data is required This function is invoked when more data is required. The subclass should place zero or more bytes into the buffers referenced by `dest`, and return a `results` value with the members set appropriately. <br> Partial success is possible. */ virtual results read( span< mutable_buffer const > dest ) = 0; }; This is not really any different from libraries that support operator<<(std::ostream&), except that the vocabulary type is buffers::sink and buffers::source instead of std::istream and std::ostream. When Boost.JSON implements these APIs, it is not "supporting Boost.Serialization" it is merely providing a wrapper for the algorithms it already has (json::parser and json::serializer) to adapt them to a common, agreed-upon interface (the buffers::sink and the buffers::source). Currently users complain that Boost is a huge bundle of libraries, and at some level this is true. But it is a bundle of libraries for a reason, because releasing the collection together as a unit allows us to ensure that all the components work together. But more importantly it allows a member library to leverage the advantages of having other member libraries available to it (Config, Variant2, mp11 come to mind). The point of implementing Boost.Buffers' sink and source interfaces in JSON is to let us leverage the benefits of the monolithic distribution model.
Boost.JSON doesn't need to depend on the implementation of the circular buffer.
Right, it wouldn't. It would only be implementing the sink and source abstract interfaces. This let's JSON participate in the growing ecosystem of Boost.Buffers enabled algorithms. I think JSON is the place for it, because the model can scale. Candidate libraries which would add value by implementing source and sink do so in the library itself. The model where a single library has to depend on an ever increasing number of other libraries in order to implement their wrappers is unsustainable. -- Regards, Vinnie Follow me on GitHub: https://github.com/vinniefalco
pt., 3 lut 2023 o 02:02 Vinnie Falco
On Thu, Feb 2, 2023 at 1:58 PM Andrzej Krzemienski
wrote: Today the basics for networking seems like good candidates for vocabulary types. Is Boost.Buffer a candidate for such a vocabulary type?
In theory, yes. The intent is for Buffers to be more general than just networking.
First question, is this library to be consumed directly by user programs, or only by other libraries, such as Boost.Beast and Boost.Http.Proto?
User programs could use it when they want to express algorithms in terms of ranges of buffers. My personal use-case is Beast, HTTP, Websocket, and JSON but this could be extended to say, base64 encoding and decoding, calculating SHA-2 digest over a range of buffers, simplified front end to things like zlib. Example:
template< class DynamicBufferOut, class DynamicBufferIn > result< void > deflate( z_params&, DynamicBufferOut& DynamicBufferIn&, bool end_of_data);
Is it possible that the final program that uses Boost.Buffer will not use Boost.ASIO?
That is possible, yes.
For me, as a consumer of networking libraries, who can afford to download and build the entire set of Boost libraries, the biggest advantage would be that Buffers would be well documented and tested.
That is an advantage yes but in my opinion the advantage is that you get access to a rich and useful set of algorithms, containers, and concepts, without also depending on a particular network implementation.
I am not convinced by the argument with Boost.JSON not depending on ASIO.
Well, users definitely do not want to drag in Asio with their JSON.
We have (I think a still unresolved issue) of Boost-serializing Boost.Optional. Should the code responsible for the serialization of Boost.Optional be part of Boost.Serialization or Boost.Optional? But we will certainly not solve it by introducing yet another library.
I think that the cases of Boost.Serialize with boost::optional, and serialization of JSON using Boost.JSON and Boost.Buffers are two completely different cases. Boost.Serialize does not define a serialization format but rather a framework for allowing the user to define serialization. Boost.JSON is something else entirely; it is a library which implements a specific protocol for describing data using structured text.
If you look at Boost.JSON you see that it has json::parser and json::serializer. These classes work a buffer at a time and they support streaming; that is, that the entire input or output need not be presented at once in order to make progress. How does Boost.Buffers help here? Well.. it provides the means for a library such as JSON, which has algorithms to produce or consume a buffer at a time, to type-erase its algorithm using the buffers::source or buffers::sink interfaces:
/// An algorithm which consumes filled buffers struct sink { virtual ~sink() = 0;
/** Called when data is available */ virtual result<void> write( span< const_buffer const > ) = 0; };
/// An algorithm which produces filled buffers struct source { /** The results of reading from the source. */ struct results { error_code ec; std::size_t bytes = 0; bool more = false; };
virtual ~source() = 0;
/** Called when more data is required
This function is invoked when more data is required. The subclass should place zero or more bytes into the buffers referenced by `dest`, and return a `results` value with the members set appropriately. <br> Partial success is possible. */ virtual results read( span< mutable_buffer const > dest ) = 0; };
This is not really any different from libraries that support operator<<(std::ostream&), except that the vocabulary type is buffers::sink and buffers::source instead of std::istream and std::ostream. When Boost.JSON implements these APIs, it is not "supporting Boost.Serialization" it is merely providing a wrapper for the algorithms it already has (json::parser and json::serializer) to adapt them to a common, agreed-upon interface (the buffers::sink and the buffers::source).
Currently users complain that Boost is a huge bundle of libraries, and at some level this is true. But it is a bundle of libraries for a reason, because releasing the collection together as a unit allows us to ensure that all the components work together. But more importantly it allows a member library to leverage the advantages of having other member libraries available to it (Config, Variant2, mp11 come to mind). The point of implementing Boost.Buffers' sink and source interfaces in JSON is to let us leverage the benefits of the monolithic distribution model.
Boost.JSON doesn't need to depend on the implementation of the circular buffer.
Right, it wouldn't. It would only be implementing the sink and source abstract interfaces. This let's JSON participate in the growing ecosystem of Boost.Buffers enabled algorithms. I think JSON is the place for it, because the model can scale. Candidate libraries which would add value by implementing source and sink do so in the library itself. The model where a single library has to depend on an ever increasing number of other libraries in order to implement their wrappers is unsustainable.
Thank you. I am convinced that extracting the Boost.Buffers library in the way you described is a good idea. Regards, &rzej;
-- Regards, Vinnie
Follow me on GitHub: https://github.com/vinniefalco
On 2/2/23 1:57 PM, Andrzej Krzemienski via Boost wrote:
Should the code responsible for the serialization of Boost.Optional be part of Boost.Serialization or Boost.Optional? But we will certainly not solve it by introducing yet another library. There are other ways to solve it, say, through optional headers.
This question comes up all the time and had never been been "solved" to everyone's satisfaction.
Regards, &rzej;
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
participants (11)
-
Alan de Freitas
-
Andrzej Krzemienski
-
Christian Mazakas
-
Gavin Lambert
-
Julien Blanc
-
Klemens Morgenstern
-
Peter Dimov
-
Richard Hodges
-
Robert Ramey
-
Ruben Perez
-
Vinnie Falco