October 2024 - Boost - lists.stage.boost.cppalliance.org

async-mqtt5 is CONDITIONALLY ACCEPTED
by Klemens Morgenstern 09 Nov '24

09 Nov '24

Hi all, thanks to everyone participating in the review! I really want to express my appreciation to the reviewers, especially those who invested multiple hours if not days to review this library: thank you Vinnie, Ruben, Mohammad and Marcelo. I interpreted Darylls review as a "weak rejection", but since that doesn't exist in the formal review process I'll count it as a rejection. Leaving the recommendations at 1 ACCEPT 3 CONDITIONAL ACCEPT 2 REJECT The actual conditions for acceptance are: 1. Correct usage of the executor. It is an asio contract that the async operation always completes on the associated executor and mqtt5 should follow this. 2. Add logging/reporting facilities. A user should be able to follow what is happening internally and ideally react to it, especially concerning reconnects. I also recommend the authors to review all the comments, including the rejections (Daryll's review had interesting details) and non-review comments such as from Peter Turcan. I would also advise that inclusion in boost is usually a time to change the name, because the library now gains the boost. prefix. I.e. boost.async-mqtt5 might be a bit cumbersome. Also note that there is no time requirement to include this library in boost in the next release. Thanks again to everyone participating in the review and to Ivica & Mireo for their submission.

15 47

Shared memory path to custom path
by Murali Kishore 08 Nov '24

08 Nov '24

Hi all, Is it possible to change the shared memory segment path from /dev/shm to other directory ex. /tmp/shm_file instead of /dev/shm/shm_file. Regards, Murali Kishore

3 4

Pretty printer module
by Дмитрий Архипов 06 Nov '24

06 Nov '24

Recently Niall Douglas posted about his idea to embed GDB pretty printers into binaries using a header. At the time I was thinking about a good way to deploy pretty printers for JSON that facilitates their autoloading. After Niall's post I experimented a little bit more, and I came to the conclusion that indeed nothing beats embedding via a C++ header in terms of convenience. At the same time I was also thinking about testing that pretty printers do in fact work. Given that pretty printers (usually) are implemented as a Python module, it initially looked like one could just run a Python program and import GDB Python modules. But actually you cannot. It turns out, though, you can run a Python script with GDB as the interpreter. After several iterations I arrived at this setup: https://github.com/boostorg/json/blob/develop/test/printers.cpp. A script converts this file into a Python script that uses GDB to break on certain lines and check if a particular expression produces a particular output when printed. While doing all this I thought that most of this work can be useful to other libraries. So, I moved the reusable parts into a separate directory (https://github.com/boostorg/json/tree/develop/pretty_printers) and added both a b2 module, and a CMake module. For B2. Importing: https://github.com/boostorg/json/blob/develop/build.jam#L8-L9 https://github.com/boostorg/json/blob/develop/build/Jamfile#L13 (Re)generating the header: https://github.com/boostorg/json/blob/develop/build/Jamfile#L88-L95 Testing: https://github.com/boostorg/json/blob/develop/test/Jamfile#L96-L101 For CMake: Importing: https://github.com/boostorg/json/blob/develop/CMakeLists.txt#L54-L55 (Re)generating the header: https://github.com/boostorg/json/blob/develop/CMakeLists.txt#L56-L63 Testing: https://github.com/boostorg/json/blob/develop/test/CMakeLists.txt#L80-L88 My questions to the community: do you find this useful? Would you use it in your Boost or non-Boost library? If you do, the module should be moved to tools rather than stay in libs/json.

12 24

[release] The 1.87.0 release will close for major changes WEDNESDAY
by Ion Gaztañaga 05 Nov '24

05 Nov '24

Calendar here: https://www.boost.org/development/ Next deadline: The master branch will close for the beta next Wednesday. — The release managers

4 4

[async-mqtt5] Formal review starts today
by Klemens Morgenstern 28 Oct '24

28 Oct '24

Hi all, the formal boost review of the async-mqtt5 library strats today, the 16th of October and will last until and including the 25th of this month. Async.MQTT5 is a professional, industrial-grade C++17 client built on Boost.Asio. This Client is designed for publishing or receiving messages from an MQTT 5.0 compatible Broker. Async.MQTT5 represents a comprehensive implementation of the MQTT 5.0 protocol standard, offering full support for publishing or receiving messages with QoS 0, 1, and 2. The library has been developed by mireo, a GPS/GIS software development company, and their CTO Ivica Siladić submitted it. The code under review is the current master, which is tagged as v1.0.2: https://github.com/mireo/async-mqtt5/releases/tag/v1.0.2 The documentation can be found here: https://spacetime.mireo.com/async-mqtt5/ Please explicitly state that you either *accept* or *reject* the inclusion of this library into boost. Also please indicate the time & effort spent on the evaluation and give the reasons for your decision. - Does this library bring real benefit to C++ developers for real world use-case? - Do you have an application for this library? - Does the API match with current best practices? - Is the documentation helpful and clear? - Did you try to use it? What problems or surprises did you encounter? - What is your evaluation of the implementation? - Do you have experience with asio? - Did you work with MQTT in the past? More information about the Boost Formal Review Process can be found at: http://www.boost.org/community/reviews.html The review is open to anyone who is prepared to put in the work of evaluating and reviewing the library. Prior experience in contributing to Boost reviews is not a requirement. Reviews can also be submitted privately to me, and I will not disclose who sent them. I might however cite passages from those reviews in my conclusion. I think I should mention that there's another mqtt library that has been proposed for boost, but has yet to find a review manager: https://github.com/redboltz/async_mqtt Some ideas of how we could review them together have been discussed on the mailing list, but this is an independent review of mireo's async-mqtt5. You don't have to consider redboltz/async_mqtt when writing your review. Thanks to Mireo & Ivica for submitting this library.

8 17

[async-mqtt5] Review ends tomorrow
by Klemens Morgenstern 27 Oct '24

27 Oct '24

Reminder: the review period for async-mqtt5 will end tomorrow on Monday the 28th. Thanks for all the great reviews received so far!

1 0

async-mqtt5 review
by Christian Mazakas 25 Oct '24

25 Oct '24

I'm going to vote that we REJECT async-mqtt5 from Boost. My primary reasoning is that so far almost all the review discussion I've seen is focusing solely on the Asio aspect which makes this a miss. I like Asio, and I've used it to deliver products as well. But I don't think this is the MQTT library for Boost. I think libraries like these are a mistake because they're so heavily focused on the ceremony of Asio that the actual meat and potatoes of the library is lost, which in this case is supposed to be MQTT parsing and serialization and broker management. I think there's a lot of utility in MQTT, but what I'd like to see in the future for libraries like these is a first-class interface _without_ I/O. For example, I'm not sure I saw any fuzzing of the underlying parsing code. A non-I/O based interface makes this trivial. I also have a lot of concerns here because the library uses Asio in its interfaces and it also internally uses Spirit. I haven't tried to compile anything but I can only imagine the binary size and the compilation times for a realistic project using this code. I don't have a lot of experience with MQTT as a protocol but my operating assumption is that if TLS/SSL can be made without I/O, so can anything else. I think exploring this design space will lead to a much better outcome for a library going into Boost. You can see prior art here via rustls, OpenSSL's BIO pairs, and Botan which uses a synchronous callback-based interface for handling things. By focusing on creating a pure protocol library, you give users the ability to add MQTT to their own I/O layers which has the extensibility and genericity I'd expect from a Boost library. It will also likely enable the library to be compiled which in this case would be a huge boon. For example, I have an asynchronous runtime which is dramatically faster than Asio and I'm capable of handling low-level networking coding but I can't use this library with my own. Instead, I'd have to roll my own MQTT support. In the ideal case, I'd be able to treat this library like I would Botan and I'd add MQTT support non-intrusively. All this being said, a protocol library that comes with an Asio-based impl is perfectly fine. My only qualm is when the library _only_ supports Asio as seems to be the case here. - Christian

8 14

[async-mqtt5] Review questions
by Ruben Perez 25 Oct '24

25 Oct '24

Hi Ivica, First of all, thanks for submitting your library to Boost. I've been reading the discussion and reference and have some questions for you: 1. The limitation listed on https://spacetime.mireo.com/async-mqtt5/async_mqtt5/auto_reconnect.html about concealing configuration issues looks concerning, as it can be very frustrating for the user. As far as I know, Boost.Redis solved it by including logging as part of its API, and Boost.MySQL did it by providing an extra diagnostics output argument, populated on operation cancellation, that informs the user of potential causes to the problem. Neither of us are very happy with our solution, to be frank. But I think it's a point that might be worth solving before going into Boost. As an option, I have been looking into abusing Asio handler tracking (https://live.boost.org/doc/libs/1_86_0/doc/html/boost_asio/overview/core/ha…) to log custom events. BOOST_ASIO_HANDLER_OPERATION might be a possibility. What are your thoughts regarding this? 2. Quoting the allocators section (https://spacetime.mireo.com/async-mqtt5/async_mqtt5/asio_compliance/allocat…) "[...] associates the allocator of the first packet in the queue to the low-level Boost.Asio function async_write on the transport layer". Does this mean that the queue might use the allocator bound to the first async_publish operation that is issued? If this is the case, wouldn't it be violating the requirement that all memory allocated by an associated allocator is deallocated before the async operation completes? 3. Quoting the executors section (https://spacetime.mireo.com/async-mqtt5/async_mqtt5/asio_compliance/executo…) "The same executor must execute mqtt_client::async_run and all the subsequent async_xxx operations". This sounds like an unusual restriction. What happens if it is violated? Wouldn't it make more sense to use the executor associated with async_run for intermediate handlers, and then the async_xxx associated executor for the final one? 4. I noticed that async_mqtt5 does not implement persistent and clean sessions. I'm no MQTT expert, so I can't evaluate whether these features are relevant or not - I just know that they exist. Is there a reason why you chose to not implement them? If so, I'd suggest adding these reasons somewhere in the docs. 5. According to what I read in the docs, message payloads might contain UTF-8 encoded text or just data blobs, depending on the payload_format_indicator property passed to publish. However, std::string is always used for payloads. As I understand it, if I want to send a binary blob, I need to copy my blob to a string (probably with a reinterpret_cast in the way), and then setting payload_format_indicator. Would it make sense to expose a simpler API taking a blob (e.g. a std::vector<unsigned char>) to make this use case safer? 6. I can't find any docs (other than the examples) on how to use TLS. >From the simplest example (https://spacetime.mireo.com/async-mqtt5/async_mqtt5/hello_world_over_tls.ht…) it looks like setting up TLS requires defining some non-trivial template specializations. I'd like these to be documented. On the other hand, the specializations seem to be using only Boost.Asio types, rather than user-defined types. What would happen if two libraries separately define these hooks because they need MQTT over TLS? Wouldn't it make more sense to provide an optional header providing these functions? 7. What's the use of the TlsContext type in mqtt_client? It looks very coupled to asio::ssl::stream. 8. I've noticed that async_receive returns with an error code whenever a reconnection occurs, which forces the user to re-subscribe to topics. This contrasts with how async_publish works, where reconnection is transparent to the user. Is there a reason for this difference? Would it make sense to implement automatic resubscription on reconnection? Regards, Ruben.

6 12

Summer Mentorship / Volunteering
by Vinnie Falco 24 Oct '24

24 Oct '24

Do any library authors or maintainers: * Desire to mentor a student, and * have the skill to mentor a student, and * have appropriate student work in one of their libraries? If yes please reply with the library and the nature of the work :) Thanks

2 1

[async-mqtt5] Mohammad's review
by Mohammad Nejati [ashtum] 23 Oct '24

23 Oct '24

Hi everyone, This is my review of the proposed async_mqtt5 library. First, I want to thank the Mireo team for submitting this library and Klemens for managing the review process. > Does this library bring real benefit to C++ developers for real world use-case? Developing a high-quality MQTT client library is no small task, and it is not something everyone wants to take on alongside their main project. Having a well-implemented and tested library would be highly valuable for C++ developers. > Do you have an application for this library? > Did you work with MQTT in the past? I have limited experience with MQTT from a previous job where I used the mqtt_cpp library to transmit and receive telemetry data in the telecommunications industry. > Does the API match with current best practices? The API simplifies library usage by providing high-level features, such as automatic reconnection, which is often essential in platforms where MQTT is used. The initiator functions adhere to Asio best practices, allowing users to provide their own executor and allocator, and include proper cancellation support. I believe there’s room to add interfaces for bulk receive and publish operations, which could enhance performance by minimizing rescheduling. This could be considered in the future based on user demand. `mqtt_client` is templated on the stream type and TLS context, which isn't inherently a problem, but it has made Boost.Beast a required dependency, which feels like a design flaw. I suggest abstracting the stream type and its required functionalities into a separate type, allowing users to just include the required implementations of it. I believe `mqtt_client` can type-erase the stream type without any measurable performance impact. All operations on the stream occur at a low frequency, so one level of indirection shouldn't matter. The only higher-frequency operations are `async_write` and `async_read_some`, but since `async_write` uses scatter/gather pattern and passes all buffers in a single call, and `async_read_some` reads into a large buffer, the number of calls to these functions will still be low. I didn’t find the `mqtt_client::brokers` interface very elegant; it could accept a range of hosts and ports and delegate the parsing of a comma-separated list to the users. > Is the documentation helpful and clear? While the documentation does a great job explaining design decisions and corner cases, I'd love to see a beginner-friendly section (like the example in the introduction) that walks through a few examples step by step. This is likely the most-read part of any library documentation and deserves more attention. > Did you try to use it? What problems or surprises did you encounter? I've tried experimenting with the provided examples on a locally run mqtt broker. The obvious problem was that there is no error reporting when things go wrong. This is a side effect of having infinite internal reconnections and no mechanism for reporting them. A logging solution or callbacks for reporting problems would be necessary. > What is your evaluation of the implementation? The implementation has a high quality and shows the authors have a very good understanding of Asio and asynchronous programing. Other than the mentioned issue with stream type, I couldn't find any significant problem during the review. I suggest specializing asio::associator instead implementation associator functions one by one: https://www.boost.org/doc/libs/develop/doc/html/boost_asio/reference/associ… Instead of using the associated allocator of the completion handler to allocate the internal queue, it might be possible to use a form of circular buffer to eliminate the need for frequent memory allocation (this might also fix the total cancellation problem in async_publish). > Do you have experience with asio? Yes, I've been a user of Asio and currently maintain Boost.Beast. > How much time did you spend on this review? I spent about 18 hours reading the documentation, reviewing some parts of the implementation, and experimenting with the interface. > Final decision and comments I vote that we conditionally accept `async_mqtt5` into Boost with the following criteria: - Break the hard dependency on Beast and TLS stream by either abstracting the stream type and placing different implementations in separate headers, or by type-erasing the stream type and constructing the connection through a factory. The type-erasure method is preferred, as it can reduce compile times and simplify usage. - Provide a mechanism for reporting connection errors and reconnection attempts to the user. - Set limits on all internal queues and allow TCP's back-pressure to regulate the speed of the producer and consumer sides. - Introduce address, UB and memory sanitizer to CI builds. Thanks, Mohammad.

3 4