April 2024 - Boost-users - lists.stage.boost.cppalliance.org

boost.asio: random_access_file, io_uring and thread safety
by Matwey V. Kornilov 30 Apr '24

30 Apr '24

Hello, random_access_file is backed by the io_uring kernel interface on Linux. However, It is still unclear from Boost.Asio documentation, whether simultaneous parallel async_write_at in-flight requests are permitted by Asio? The queue of simultaneous requests is natively supported by io_uring. Is it still true for Asio? Is it also safe to fire parallel async_write_at from different threads? -- With best regards, Matwey V. Kornilov

5 7

Is there a buffered tcp stream in ASIO?
by accelerator0099 29 Apr '24

29 Apr '24

|Prepare a tcp server listening on [::]:23380, run the following code: #include <boost/asio.hpp> namespace io = boost::asio; using boost::system::system_error; int main() { try { io::io_context ctx; using io::ip::tcp; using io::buffer; tcp::socket s1(ctx); s1.connect(tcp::endpoint(io::ip::address_v4::loopback(), 23380)); for (size_t i{}; i != 10; ++i) io::write(s1, buffer("123", 3)); } catch (system_error& e) { return 1; } } strace says there are 10 syscalls: connect(6, {sa_family=AF_INET, sin_port=htons(23380), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 sendto(6, "123", 3, MSG_NOSIGNAL, NULL, 0) = 3 If I use boost::asio::ip::tcp::iostream: #include <boost/asio.hpp> namespace io = boost::asio; using boost::system::system_error; int main() { try { io::io_context ctx; using io::ip::tcp; using io::buffer; tcp::iostream s1{ tcp::socket(ctx) }; auto& nl = s1.socket(); nl.connect(tcp::endpoint(io::ip::address_v4::loopback(), 23380)); for (size_t i{}; i != 10; ++i) s1.write("123", 3); s1.flush(); } catch (system_error& e) { return 1; } } It does not make sense, still 10 syscalls: connect(6, {sa_family=AF_INET, sin_port=htons(23380), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 ioctl(6, FIONBIO, [1]) = 0 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="123", iov_len=3}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 3 Is there a stream in asio that could buffer the multiple and small pieces of data into one syscall? |

2 2

Difference in regex_replace replacement format handling between Boost.Regex and std::regex
by Josué Andrade Gomes 22 Apr '24

22 Apr '24

There is a difference in how the replacement format string is handled by regex_replace in Boost.Regex and std::regex. Consider the following example: #include <iostream>#include <regex>#include <boost/regex.hpp> int main() { std::string s{"a:b:c"}; std::string rx{":"}; std::string r{"\\$&"}; // Using std::regex std::string result_std = std::regex_replace(s, std::regex(rx), r); std::cout << "std::regex result: " << result_std << std::endl; // Outputs "a\:b\:c" // Using Boost.Regex std::string result_boost = boost::regex_replace(s, boost::regex(rx), r); std::cout << "Boost.Regex result: " << result_boost << std::endl; // Outputs "a$&b$&c" return 0; } std::regex result: a\:b\:c Boost.Regex result: a$&b$&c Affects Boost.Regex 1.84. I didn’t try other versions. Tested with GCC 13.2, clang 18.10 , cl 19.39.33523 and results are the same. Not tested with other versions. https://github.com/boostorg/regex/issues/210

3 2

Regarding a Vulnerability in Boost.Regex library
by kamallochan Jena 18 Apr '24

18 Apr '24

Hello everyone, Hope you all are doing well. There is a Vulnerability reported on Boost library as mentioned below. Any guidance or assistance or reply to this mail would be greatly appreciated. *Vulnerability ID:* BDSA-2018-2656 *Vulnerability Details:* Boost has a flaw in the function boost::re_detail_NUMBER::basic_regex_creator which can lead to a buffer over-read. An attacker can craft and send a malicious file which will trigger the buffer over-read, leading to a denial-of-service. Few query w.r.t boost::re_detail_NUMBER::basic_regex_creator() function: 1. Does Boost.Regex library or any Boost library internally use this function? 2. If the answer is yes, Which all libraries use this function? 3. Is this a known vulnerability and is it fixed in the latest Boost version? please provide some insights like (any change list or file name etc). Our project uses a boost library(which has file "boost/regex/v4/basic_regex_creator.hpp") and uses Boost.Regex library functions. *Should we be worried about this vulnerability (BDSA-2018-2656) affecting our code? * Thanks!

3 4

Any idea on CWE-704(BDSA-2018-1263) and CWE-126 (BDSA-2018-2656)
by kamallochan Jena 15 Apr '24

15 Apr '24

Dear all, Could someone assist me in determining if any version of Boost library is vulnerable to the following CWE? BDSA-2018-1263 CWE-704 and BDSA-2018-2656 CWE-126 Additionally, I would appreciate any details regarding these vulnerabilities, such as - the specific version of the Boost library where the vulnerability was identified - the name of the source code file, the exact line of code and function that causes the vulnerability - Whether the latest version of Boost includes a fix, and if so, which version of the library it was addressed in. Any guidance or assistance on this matter would be greatly appreciated. Thanks, Kjena

2 2

Re: [Boost-users] [beast] I can't write a hello world server
by Senthil Cheetancheri 10 Apr '24

10 Apr '24

(Trying to repost because the previous message was rejected due to size limits.) Hi Loup, The reason I'm taking the time to ask here is because I feel there is something fundamental, likely obvious to the experts, that I don't understand about how beast (and, I'm guessing, asio) works, and how I'm supposed to use it. It seems like you are quite comfortable with C++. That is good because it is a necessary background for this. Boost.Asio and .Beast are inherently advanced topics. The examples use quite advanced C++ knowledge. It could appear simple to experts who understand asio and C++, as well as to newcomers if they were lucky to get the right orientation when they began to learn C++. I had to unlearn a lot of 'C' and old C++ notions. Here are some resources that helped me reach the Aha! moment after a lot of struggles. They are all well worth the patient time you will need. (They are from my bookmarks and are a bit dated. You can easily get the corresponding latest versions.) 1. ASIO: https://www.boost.org/doc/libs/1_79_0/doc/html/boost_asio.html 2. Beast: Watch this video https://youtu.be/7FQwAjELMek at the bottom of the tutorials page https://www.boost.org/doc/libs/1_80_0/libs/beast/doc/html/beast/examples.ht…. This video was a turning point for me. 3. If you would like interactive help, join the #beast channel of the cpplang slack https://cppalliance.org/slack/. They are very kind and super helpful. When you get asio and beast to work well, they are beautiful and give you joy. HTH. Thanks, Senthil.

3 3

Boost.ASIO C++ 20 coroutine equivalent of PPL task_completion_event or JavaScript Promise.resolve?
by Willem Mitchell 09 Apr '24

09 Apr '24

I'm trying to write a simple ASIO send/receiver that uses message queueing. The goal is to have a number of virtual streams over a single TCP connection. This is pretty simple using callbacks, but I can't figure out the coroutines alternative. Here is my simplified sample code. Note that I'm using asio::experimental::promise the same way I would use std::promise which obviously doesn't work, but it illustrates what I'm trying to do. What is the simplest solution to my problem? #pragma once #include "asio.hpp" #include "asio/co_spawn.hpp" #include "asio/detached.hpp" #include "asio/experimental/co_composed.hpp" #include "asio/experimental/promise.hpp" #include "asio/use_awaitable.hpp" #include <coroutine> #include <cstdint> #include <map> #include <memory> #include <queue> struct Message { uint64_t id; uint64_t streamId; uint64_t priority; }; class NetworkConnection { public: NetworkConnection(asio::io_context &ioContext) : ioContext_(ioContext), sendTimer_(ioContext_), receiveTimer_(ioContext_){}; asio::awaitable<std::unique_ptr<Message>> receive(uint64_t streamId) { auto promise = std::make_shared< asio::experimental::promise<std::unique_ptr<Message>>>(); auto future = promise->get_future(); receives_[streamId] = promise; if (!receiving_) { receiving_ = true; co_spawn(ioContext_, startReceiving(), asio::detached); } co_return co_await future; } asio::awaitable<void> send(std::unique_ptr<Message> message) { auto promise = std::make_shared<asio::experimental::promise<void>>(); auto future = promise->get_future(); sends_[message->id] = promise; if (!sending_) { sending_ = true; co_spawn(ioContext_, startSending(), asio::detached); } co_await future; co_return; } private: asio::io_context &ioContext_; asio::steady_timer receiveTimer_; asio::steady_timer sendTimer_; std::map<uint64_t, std::shared_ptr< asio::experimental::promise<std::unique_ptr<Message>>>> receives_; std::map<uint64_t, std::shared_ptr<asio::experimental::promise<void>>> sends_; std::queue<std::unique_ptr<Message>> sendQueue_; bool receiving_ = false; bool sending_ = false; void completeSend(uint64_t messageId) { auto it = sends_.find(messageId); if (it != sends_.end()) { it->second->set_value(); // This resumes the send operation sends_.erase(it); // Clean up } } void completeReceive(std::unique_ptr<Message> message) { auto it = receives_.find(message->streamId); if (it != receives_.end()) { it->second->set_value( std::move(message)); // This resumes the send operation sends_.erase(it); // Clean up } } asio::awaitable<void> startReceiving() { while (!receives_.empty()) { // Simulate receiving a message sendTimer_.expires_after(std::chrono::seconds(1)); co_await sendTimer_.async_wait(asio::use_awaitable); auto message = std::make_unique<Message>(); message->id = 1; message->streamId = 1; message->priority = 1; completeReceive(std::move(message)); } receiving_ = false; co_return; } asio::awaitable<void> startSending() { while (!sends_.empty()) { sendTimer_.expires_after(std::chrono::seconds(1)); auto message = std::move(sendQueue_.front()); sendQueue_.pop(); co_await sendTimer_.async_wait(asio::use_awaitable); completeSend(message->id); } sending_ = false; co_return; } }; class Stream { public: Stream(std::shared_ptr<NetworkConnection> connection) : connection_(std::move(connection)) {} asio::awaitable<std::unique_ptr<Message>> receive() { co_return co_await connection_->receive(id_); } asio::awaitable<void> send(std::unique_ptr<Message> message) { message->streamId = id_; co_return co_await connection_->send(std::move(message)); } private: uint64_t id_; std::shared_ptr<NetworkConnection> connection_; };

1 0

Re: [Boost-users] [beast] I can't write a hello world server
by Mohammad Nejati [ashtum] 08 Apr '24

08 Apr '24

> Then I wait almost 4 seconds for a successful compilation. `-O2` and `-Og -g` are worse at 6 seconds. You are right, Asio and Asio based libraries are usually heavily templated as they need to work with generic executors, buffers and stream types. There are new facilities in Asio for alleviating this issue like any_executor and any_completion_handler which you can utilize to implement async operations in a source file. > I figured I'd write a wrapper, that would confine boost/asio/beast includes to a single compilation unit. The API I'm aiming for looks like this I think you might just use `beast::http::request<string_body>` and `beast::http::response<string_body>` types, you don't need to include all header files, you can just include `boost/beast/http/message.hpp`. > And yet I can sense a potential problem right off the bat: in the `http_server()` function, we call `std::move(socket)`. This move seems to be mandatory here, because removing triggers a compilation error. My point is, it looks like upon incoming connection we move the socket out of its outer context, such that it is no longer bound to the IO context. Is this another error in the example? You don't need to pass a socket to acceptor, it has multiple overloads: https://www.boost.org/doc/libs/1_84_0/doc/html/boost_asio/reference/basic_s… One of them creates a socket and passes it to the completion token, so, you can use it like: acceptor.async_accept([](auto ec, auto socket){ // ... }); Regards, Mohammad On Thu, Apr 4, 2024 at 4:50 PM Loup VAILLANT via Boost-users <boost-users(a)lists.boost.org> wrote: > > Hello, > > So this is rather embarrassing. After working more than 2 days full time I can't even get the most basic code running. I've scoured the docs and tried to find some tutorial, no luck. My onboarding experience so far has been… less than perfect. > > The reason I'm taking the time to ask here is because I feel there is something fundamental, likely obvious to the experts, that I don't understand about how beast (and, I'm guessing, asio) works, and how I'm supposed to use it. Which is probably why seemingly innocuous changes to official example code are causing memory corruptions left and right. > > Let's start at the beginning. > > My first attempt with beast was getting this code to work: > https://www.boost.org/doc/libs/1_73_0/libs/beast/example/http/server/small/… > > Compiling it with `g++ -std=c++20`, gets me a compilation error, thankfully easily fixed. > Then I wait almost 4 seconds for a successful compilation. `-O2` and `-Og -g` are worse at 6 seconds. > > I know many developers would say a few seconds are no big deal (though wen we get to XKCD sword fights it kind of is), but my problem here is that I didn't want to wait every time I made a business logic change to my code. But since 95% of compilation time came from including a huge header-only library, I figured I'd write a wrapper, that would confine boost/asio/beast includes to a single compilation unit. The API I'm aiming for looks like this: > > struct Http_message { > std::unordered_map<std::string, std::string> headers; > std::string body; > }; > struct Http_request : public Http_message { > std::string method; > std::string target; > }; > struct Http_response : public Http_message { > int status; > }; > typedef std::function<Http_response(const Http_request &request)> response_cb; > void http_start(unsigned short port, response_cb on_respond); > > (Note 1: I want web sockets as well, but one problem at a time) > (Note 2: I tried wrapping around http::request and http::response more directly to avoid unnecessary copies, but I ran into lifetime issues.) > > To its credit, this small server example seemed to run well enough, and could serve as a starting point. Before I untangle the business code from the HTTP specific stuff though, I needed to simplify this example a bit. So *this* is my actual starting point: > > > #include <boost/beast/core.hpp> > #include <boost/beast/http.hpp> > > namespace beast = boost::beast; > namespace http = beast::http; > namespace net = boost::asio; > using tcp = boost::asio::ip::tcp; > > class http_connection : public std::enable_shared_from_this<http_connection> > { > public: > http_connection(tcp::socket socket) : socket_(std::move(socket)){} > > void start() > { > auto self = shared_from_this(); > http::async_read( > socket_, buffer_, request_, > [self](beast::error_code ec, std::size_t) { > if(!ec) { self->process_request(); } > }); > deadline_.async_wait( > [self](beast::error_code ec) { > if(!ec) { self->socket_.close(ec); } > }); > } > > private: > tcp::socket socket_; > beast::flat_buffer buffer_{8192}; > http::request <http::dynamic_body> request_; > http::response<http::dynamic_body> response_; > net::steady_timer deadline_ { > socket_.get_executor(), std::chrono::seconds(60) }; > > void process_request() { > response_.version(request_.version()); > response_.keep_alive(false); > response_.result(http::status::ok); > response_.set(http::field::server, "Beast"); > response_.set(http::field::content_type, "text/plain"); > beast::ostream(response_.body()) > << "Target = " << request_.target() << "\n" > << "Method = " << std::string(request_.method_string()) << "\n"; > response_.set(http::field::content_length, > std::to_string(response_.body().size())); > > auto self = shared_from_this(); > http::async_write( > socket_, response_, > [self](beast::error_code ec, std::size_t) { > self->socket_.shutdown(tcp::socket::shutdown_send, ec); > self->deadline_.cancel(); > }); > } > }; > > void http_server(tcp::acceptor& acceptor, tcp::socket& socket) > { > auto cb = [&](beast::error_code ec) { > if(!ec) {std::make_shared<http_connection>(std::move(socket))->start();} > http_server(acceptor, socket); > }; > acceptor.async_accept(socket, cb); > } > > int main(void) > { > auto const address = net::ip::make_address("0.0.0.0"); > unsigned short port = 8080; > net::io_context ioc {1}; > tcp::acceptor acceptor{ioc, {address, port}}; > tcp::socket socket {ioc}; > http_server(acceptor, socket); > ioc.run(); > return 0; > } > > > So far it works. It listens to port 8080, and a curl request or 3 to the server works as expected: > > > $ curl -XYOLO -v http://localhost:8080/foobar > * Trying 127.0.0.1:8080... > * Connected to localhost (127.0.0.1) port 8080 (#0) > > YOLO /foobar HTTP/1.1 > > Host: localhost:8080 > > User-Agent: curl/7.81.0 > > Accept: */* > > > * Mark bundle as not supporting multiuse > < HTTP/1.1 200 OK > < Connection: close > < Server: Beast > < Content-Type: text/plain > < Content-Length: 31 > < > Target = /foobar > Method = YOLO > * Closing connection 0 > > And yet I can sense a potential problem right off the bat: in the `http_server()` function, we call `std::move(socket)`. This move seems to be mandatory here, because removing triggers a compilation error. My point is, it looks like upon incoming connection we move the socket out of its outer context, such that it is no longer bound to the IO context. Is this another error in the example? > > Okay then, how about I pass the socket all the way by reference instead? Let the constructor get a `tcp::socket&`, store this in a `tcp::socket&` member variable, remove all the `std::move`. It compiles, it runs the first query, but at the second query my program just freezes. Or loops, I don't know. > > So that's the first thing. I am _really confused_ about how this socket is supposed to work. Why does it block when I remove the standard moves? Isn't it supposed to be the same socket bound to the same IO context all the way? And I don't think it falls out of scope before the program ends either, it lives on the stack of the `main()` function, and the `ioc.run()`we call at the end blocks until we're done with the async reads (which is, never). > > **Can someone explain to me what the hell is going on with this socket?** > > Right now though I can accept that some dark secret lies behind this `std::move()`, and leave it alone for now. > > Moving on, I'll just admit for the time being that something magical and forbidden lies between this mysterious `std::move()` and prudently leave it behind. Now it's working again, and I can move (no pun intended) to the next step: extracting the business logic to an external function. I start with a trivial extraction: > > void respond(http::response<http::dynamic_body> &response, > const http::request <http::dynamic_body> &request) > { > response.result(http::status::ok); > response.set(http::field::content_type, "text/plain"); > beast::ostream(response.body()) > << "Target = " << request.target() << "\n" > << "Method = " << std::string(request.method_string()) << "\n"; > } > > And modify `process_request()` accordingly: > > void process_request() { > response_.version(request_.version()); > response_.keep_alive(false); > response_.set(http::field::server, "Beast"); > respond(response_, request_); > response_.set(http::field::content_length, > std::to_string(response_.body().size())); > > auto self = shared_from_this(); > http::async_write( > socket_, response_, > [self](beast::error_code ec, std::size_t) { > self->socket_.shutdown(tcp::socket::shutdown_send, ec); > self->deadline_.cancel(); > }); > } > > That works, but this is not quite enough: my goal here is to isolate my user code from the overly long compilation times imposed by this huge boost::beast. Maybe I can include the request & response headers without pulling in the kitchen sink with it, but I don't quite trust it. So let's wrap the `http::response` in some class so I can keep my headers nice and small: > > > class http_response { > public: > http_response(http::response<http::dynamic_body> &response) > : response_(response) {} > > void set_status(int status) { > response_.result(status); > } > void set_header(const std::string &title, > const std::string &value) { > response_.set(title, value); > } > void set_body(const std::string &body) { > beast::ostream(response_.body()) << body; > } > > private: > http::response<http::dynamic_body> &response_; > }; > > > Nothing fancy. And with a forward declaration I should be able to avoid including the response in the header that defines that class. Now I need to modify my business code accordingly: > > > void respond(http_response &response, > const http::request<http::dynamic_body> &request) > { > response.set_status(200); > response.set_header("Content-Type", "text/plain"); > response.set_body( > "Target = " + std::string(request.target()) + "\n" + > "Method = " + std::string(request.method_string()) + "\n"); > } > > > Still not free from `http::request <http::dynamic_body>` yet, but one step at a time. And finally, I need to adjust `process_request()`: > > void process_request() { > response_.version(request_.version()); > response_.keep_alive(false); > response_.set(http::field::server, "Beast"); > http_response response(response_); > respond(response, request_); > response_.set(http::field::content_length, > std::to_string(response_.body().size())); > > auto self = shared_from_this(); > http::async_write( > socket_, response_, > [self](beast::error_code ec, std::size_t) { > self->socket_.shutdown(tcp::socket::shutdown_send, ec); > self->deadline_.cancel(); > }); > } > > > Huh, what do you know, this time around it worked. I no longer experience the lifetime issues I've once had. Okay, I guess I'll add the request wrapper, then: > > > class http_request { > public: > http_request(const http::request<http::dynamic_body> &request) > : request_(request) {} > > std::string target() const { return std::string(request_.target()); } > std::string method() const { return std::string(request_.method_string()); } > > private: > const http::request<http::dynamic_body> &request_; > }; > > void respond(http_response &response, const http_request &request) > { > response.set_status(200); > response.set_header("Content-Type", "text/plain"); > response.set_body( > "Target = " + request.target() + "\n" + > "Method = " + request.method() + "\n"); > } > > void process_request() { > response_.version(request_.version()); > response_.keep_alive(false); > response_.set(http::field::server, "Beast"); > http_response response(response_); > http_request request (request_); > respond(response, request); > response_.set(http::field::content_length, > std::to_string(response_.body().size())); > > auto self = shared_from_this(); > http::async_write( > socket_, response_, > [self](beast::error_code ec, std::size_t) { > self->socket_.shutdown(tcp::socket::shutdown_send, ec); > self->deadline_.cancel(); > }); > } > > > And it still works! Maybe it'll work out in the end? > Regardless, thanks for the rubber ducking I guess. > > My first question remains though: what the hell is going on with the socket? Why can't I just pass a reference along? Why do we even need a move constructor? Is there some special semantics, or `boost::asio` specific arcane, I should be privy to? > > Loup. > > _______________________________________________ > Boost-users mailing list > Boost-users(a)lists.boost.org > https://lists.boost.org/mailman/listinfo.cgi/boost-users

3 2