Proposal for Inclusion of Async.MQTT5 Library in Boost

Ivica Siladic

30 Nov 2023 30 Nov '23

3:05 p.m.

Dear Boost Developers, we are writing to introduce a new C++ Async.MQTT5 library that we believe would be a valuable addition to Boost. MQTT has become the de-facto standard for IoT communication, with billions of IoT devices relying on it to publish or retrieve data from the cloud. Despite its widespread use, integrating MQTT into a C++ embedded application remains a complex and error-prone task. This challenge primarily arises from the absence of a higher-level library that offers a straightforward interface for message publishing and receiving while abstracting MQTT protocol intricacies and effectively managing various corner cases inherent in MQTT implementation. The primary objective behind developing Async.MQTT5 was to address these challenges comprehensively, allowing developers to focus on the messages they need to send without concerns about underlying network connectivity issues. Async.MQTT5 is a C++20 client implementation of the MQTT 5.0 protocol, leveraging Boost Asio and Boost Spirit. It fully adheres to the Asio asynchronous model, providing comprehensive support for all forms of asynchronous completion tokens, per-operation cancellation, and custom allocators. The library is designed to work seamlessly with any ordered network transport, typically TCP, SSL/TLS over TCP, or WebSockets (secured and unsecured). Our goal with Async.MQTT5 is to offer developers a simple and clear C++ interface for sending and receiving messages using the MQTT 5.0 protocol. Emphasizing code readability was a key design principle of Async.MQTT5. For instance, tedious protocol message parsing/composing is implemented declaratively (partially utilizing Boost Spirit). Additionally, Asio composed operations are implemented using a novel, clear, and highly efficient "chaining functional operators" approach. It is worth noting that the library is already written following Boost style guidelines, and the documentation is crafted and formatted in line with Boost conventions. You can find the library at https://github.com/mireo/async-mqtt5, and the documentation is available at https://spacetime.mireo.com/async-mqtt5/. We highly value your opinion and would appreciate any feedback you can provide on the Async.MQTT5 library. Thank you for your time and consideration. Best regards, Ivica Siladic Bruno Iljazovic Korina Simicevic

Show replies by date

Vinnie Falco

30 Nov 30 Nov

3:55 p.m.

On Thu, Nov 30, 2023 at 7:24 AM Ivica Siladic via Boost < boost@lists.boost.org> wrote:

...

we are writing to introduce a new C++ Async.MQTT5 library that we believe would be a valuable addition to Boost.

I endorse this library for review. Thanks

Richard Hodges

4 p.m.

seconded On Thu, 30 Nov 2023 at 16:55, Vinnie Falco via Boost wrote:

...

On Thu, Nov 30, 2023 at 7:24 AM Ivica Siladic via Boost < boost@lists.boost.org> wrote:

...
we are writing to introduce a new C++ Async.MQTT5 library that we believe would be a valuable addition to Boost.

I endorse this library for review.

Thanks

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Andrzej Krzemienski

1 Dec 1 Dec

4:25 p.m.

czw., 30 lis 2023 o 16:24 Ivica Siladic via Boost napisał(a):

...

Dear Boost Developers,

we are writing to introduce a new C++ Async.MQTT5 library that we believe would be a valuable addition to Boost.

MQTT has become the de-facto standard for IoT communication, with billions of IoT devices relying on it to publish or retrieve data from the cloud. Despite its widespread use, integrating MQTT into a C++ embedded application remains a complex and error-prone task. This challenge primarily arises from the absence of a higher-level library that offers a straightforward interface for message publishing and receiving while abstracting MQTT protocol intricacies and effectively managing various corner cases inherent in MQTT implementation.

The primary objective behind developing Async.MQTT5 was to address these challenges comprehensively, allowing developers to focus on the messages they need to send without concerns about underlying network connectivity issues.

Async.MQTT5 is a C++20 client implementation of the MQTT 5.0 protocol, leveraging Boost Asio and Boost Spirit. It fully adheres to the Asio asynchronous model, providing comprehensive support for all forms of asynchronous completion tokens, per-operation cancellation, and custom allocators. The library is designed to work seamlessly with any ordered network transport, typically TCP, SSL/TLS over TCP, or WebSockets (secured and unsecured).

Our goal with Async.MQTT5 is to offer developers a simple and clear C++ interface for sending and receiving messages using the MQTT 5.0 protocol.

Emphasizing code readability was a key design principle of Async.MQTT5. For instance, tedious protocol message parsing/composing is implemented declaratively (partially utilizing Boost Spirit). Additionally, Asio composed operations are implemented using a novel, clear, and highly efficient "chaining functional operators" approach.

It is worth noting that the library is already written following Boost style guidelines, and the documentation is crafted and formatted in line with Boost conventions.

You can find the library at https://github.com/mireo/async-mqtt5, and the documentation is available at https://spacetime.mireo.com/async-mqtt5/.

We highly value your opinion and would appreciate any feedback you can provide on the Async.MQTT5 library.

Thank you for your time and consideration.

Thank you for submitting this proposal. But let me be the devil's advocate here. I did not look at the implementation, or documentation in detail. My concern is about "adding an ASIO-based client for a protocol". I am not familiar with MQTT 5.0 and this is the first time I have seen this name. My impression is that it is not popular, surely far less than the HTTP protocol. My question to the Boost community is, does every good library need to be included in Boost? I thought that Boost was supposed to be an "extended Standard Library", containing: 1. Portability fixes 2. Implementing missing language features 3. Vocabulary types (I consider Boost.ASIO one) 4. Frequently used components 5. Ground-breaking designs and ideas Async.MQTT5 doesn't fit into any of these categories. (Neither does Boost.AEDIS or Boost.MySql.) Do we have a clear set of criteria for what belongs to Boost? And I do not mean only the quality bar, but also the range of business domain. What's wrong with a good library hosted on its own in GitHub? Regards, &rzej;

Vinnie Falco

5:42 p.m.

On Fri, Dec 1, 2023 at 8:25 AM Andrzej Krzemienski via Boost wrote:

...

My impression is that it is not popular, surely far less than the HTTP protocol.

Nothing is going to come close to the popularity of HTTP but this MQTT is a super-annoying protocol which unfortunately has a dedicated niche for which the use-case is ideal. People invent their own MQTT libraries almost as often as they write their own JSON libraries.

...

My question to the Boost community is, does every good library need to be included in Boost?

In general, I would say no.

...

I thought that Boost was supposed to be an "extended Standard Library", containing:

1. Portability fixes 2. Implementing missing language features 3. Vocabulary types (I consider Boost.ASIO one) 4. Frequently used components 5. Ground-breaking designs and ideas

Async.MQTT5 doesn't fit into any of these categories. (Neither does Boost.AEDIS or Boost.MySql.)

I agree that Aedis, MySQL, and MQTT5 do not precisely fit your list above. However, there is a certain synergy with Asio, Beast, Aedis, MySQL, and the proposed MQTT which has merit. The reality is that users are increasingly turning away from Boost in favor of the standard library (this is a conversation for another thread). If Boost only limits itself to items one through four in your list above, the value proposition for Boost becomes unclear over time. One of the ways we might envision Boost reinventing itself is to capitalize on the one area where Boost is so far ahead that there is simply no competition: "The C++ standard can't even connect to the Internet." - Vinnie Falco Based on the papers coming out of wg21 I suspect that committee-designed networking and its accompanying replacement for Kohlhoff's executors are going to be at the same level of quality and usability as, say, ranges. Make of that what you will. In the meanwhile if we can position Boost to have a strong set of networking libraries that work together well this gives us a unique advantage. People will grudgingly download Boost despite the usual irrational complaints ("too big", "too many dependencies", "inconsistent quality") because they cannot get this set of harmonious features anywhere else. There is a group of authors and contributors working on a technology demo which integrates all of the networking libraries in Boost into several applications to showcase how the libraries might be used together to build enterprise-ready systems. This project is called "Boost ServerTech" and you can learn more about it here: BoostServerTech project proposal https://docs.google.com/document/d/1ZQrod1crs8EaNLLqSYIRMacwR3Rv0hC5l-gfL-jO... Architecture Documentation https://anarthal.github.io/servertech-chat/01-architecture.html Boost ServerTech Chat https://github.com/anarthal/servertech-chat Ruben Perez is a C++ Alliance Staff Engineer who is spearheading this project.

...

Do we have a clear set of criteria for what belongs to Boost? And I do not mean only the quality bar, but also the range of business domain. What's wrong with a good library hosted on its own in GitHub?

There is not a clear set of criteria, but I think that our Contributor Guide should offer some kind of guidance on "what belongs in Boost." A rigid set of rules doesn't really make sense for me. Since libraries are quite unique as well as their authors, we should evaluate each library on a case by case basis, with some general principles posted in the documentation so that newcomers to the mailing list can orient themselves. Thanks

Richard Hodges

5:47 p.m.

...

...
My question to the Boost community is, does every good library need to be included in Boost?

In general, I would say no.

In contrast, I would say yes. I have always wanted to be able to compile any possible program by merely linking with a single massive library. Dependency management is hell. MFC was horrid, but at least you could write a useful program that included a UI and sound with it. The standard library will clearly never serve this role.

Christian Mazakas

7:28 p.m.

There was a huge onus to accept Boost.MySQL because of its licensing. libmysqlclient is GPL'd where Boost.MySQL proposed a permissive license which did mostly the exact same stuff. To that end, Boost.MySQL is a welcome addition to Boost because it essentially sets in stone a quality client implementation of the MySQL protocol without users being hampered by the GPL. The other libraries such as this? It's less so. There's often multiple libraries already being used in production that aren't as restrictively licensed which means there's less onus for accepting them. In general, I'd defer to users in this regard. If Boost users think it's useful to bundle all this up, I'd say accept but without that, I'm cautious about this wave of web-based tech hitting Boost. - Christian

Andrzej Krzemienski

3 Dec 3 Dec

5:42 p.m.

pt., 1 gru 2023 o 20:30 Christian Mazakas via Boost napisał(a):

...

There was a huge onus to accept Boost.MySQL because of its licensing.

libmysqlclient is GPL'd where Boost.MySQL proposed a permissive license which did mostly the exact same stuff.

To that end, Boost.MySQL is a welcome addition to Boost because it essentially sets in stone a quality client implementation of the MySQL protocol without users being hampered by the GPL.

So, I understand there was a need to have a MySQL client with the Boost Software License (BSL). But it still isn't obvious that this should imply making this library part of Boost. We have good libraries on the Internet, such as SOCI https://github.com/SOCI/soci, distributed under BSL, and not being part of Boost.

...

The other libraries such as this? It's less so. There's often multiple libraries already being used in production that aren't as restrictively licensed which means there's less onus for accepting them.

In general, I'd defer to users in this regard. If Boost users think it's useful to bundle all this up, I'd say accept but without that, I'm cautious about this wave of web-based tech hitting Boost.

I am not sure how this would work. If nothing else, users' expectations are often contradictory. Different users want different things.Some users only want a distribution system, and since there isn't any for C++, they would overuse Boost. Is 10 users wanting to have a library in enough to let the library in? Regards, &rzej;

Espen Harlinn

6:17 p.m.

...

...
Andrzej Krzemienski: Is 10 users wanting to have a library in enough to let the library in?

My guess is that there will be quite a bit more than that. I would certainly appreciate support for MQTT, and other messaging protocols like AMQP, in boost. Best regards Espen Harlinn -----Original Message----- From: Boost On Behalf Of Andrzej Krzemienski via Boost Sent: Sunday, 3 December 2023 18.43 To: boost@lists.boost.org Cc: Andrzej Krzemienski Subject: Re: [boost] Proposal for Inclusion of Async.MQTT5 Library in Boost pt., 1 gru 2023 o 20:30 Christian Mazakas via Boost napisał(a):

...

There was a huge onus to accept Boost.MySQL because of its licensing.

libmysqlclient is GPL'd where Boost.MySQL proposed a permissive license which did mostly the exact same stuff.

To that end, Boost.MySQL is a welcome addition to Boost because it essentially sets in stone a quality client implementation of the MySQL protocol without users being hampered by the GPL.

...

The other libraries such as this? It's less so. There's often multiple libraries already being used in production that aren't as restrictively licensed which means there's less onus for accepting them.

In general, I'd defer to users in this regard. If Boost users think it's useful to bundle all this up, I'd say accept but without that, I'm cautious about this wave of web-based tech hitting Boost.

Andrzej Krzemienski

6:11 p.m.

pt., 1 gru 2023 o 18:42 Vinnie Falco napisał(a):

...

...
My impression is that it is not popular, surely far less than the HTTP

On Fri, Dec 1, 2023 at 8:25 AM Andrzej Krzemienski via Boost wrote: protocol.

Nothing is going to come close to the popularity of HTTP but this MQTT is a super-annoying protocol which unfortunately has a dedicated niche for which the use-case is ideal. People invent their own MQTT libraries almost as often as they write their own JSON libraries.

...
My question to the Boost community is, does every good library need to be included in Boost?

In general, I would say no.

...
I thought that Boost was supposed to be an "extended Standard Library", containing:

1. Portability fixes 2. Implementing missing language features 3. Vocabulary types (I consider Boost.ASIO one) 4. Frequently used components 5. Ground-breaking designs and ideas

Async.MQTT5 doesn't fit into any of these categories. (Neither does Boost.AEDIS or Boost.MySql.)

I agree that Aedis, MySQL, and MQTT5 do not precisely fit your list above. However, there is a certain synergy with Asio, Beast, Aedis, MySQL, and the proposed MQTT which has merit. The reality is that users are increasingly turning away from Boost in favor of the standard library (this is a conversation for another thread). If Boost only limits itself to items one through four in your list above, the value proposition for Boost becomes unclear over time.

Boost used to be a testing and experimentation ground before LWG. Are you saying LWG doesn't need new libraries? Or that the community doesn't need standard-library-like libraries?

...

One of the ways we might envision Boost reinventing itself is to capitalize on the one area where Boost is so far ahead that there is simply no competition:

"The C++ standard can't even connect to the Internet." - Vinnie Falco

"Reinventing"? It sounds like "we are a group of people, one adventure ended, let's look for a new one". Is that what you mean?

...

Based on the papers coming out of wg21 I suspect that committee-designed networking and its accompanying replacement for Kohlhoff's executors are going to be at the same level of quality and usability as, say, ranges. Make of that what you will.

In the meanwhile if we can position Boost to have a strong set of networking libraries that work together well

So, are Beast, AEDIS, MySql and now this MQTT5 working well together? Do they use similar structure, naming conventions and the design? this gives us a unique

...

advantage.

Advantage over what? Regards, &rzej; People will grudgingly download Boost despite the usual

...

irrational complaints ("too big", "too many dependencies", "inconsistent quality") because they cannot get this set of harmonious features anywhere else.

There is a group of authors and contributors working on a technology demo which integrates all of the networking libraries in Boost into several applications to showcase how the libraries might be used together to build enterprise-ready systems. This project is called "Boost ServerTech" and you can learn more about it here:

BoostServerTech project proposal

https://docs.google.com/document/d/1ZQrod1crs8EaNLLqSYIRMacwR3Rv0hC5l-gfL-jO...

Architecture Documentation https://anarthal.github.io/servertech-chat/01-architecture.html

Boost ServerTech Chat https://github.com/anarthal/servertech-chat

Ruben Perez is a C++ Alliance Staff Engineer who is spearheading this project.

...
Do we have a clear set of criteria for what belongs to Boost? And I do

not

...
mean only the quality bar, but also the range of business domain. What's wrong with a good library hosted on its own in GitHub?

There is not a clear set of criteria, but I think that our Contributor Guide should offer some kind of guidance on "what belongs in Boost." A rigid set of rules doesn't really make sense for me. Since libraries are quite unique as well as their authors, we should evaluate each library on a case by case basis, with some general principles posted in the documentation so that newcomers to the mailing list can orient themselves.

Thanks

Vinnie Falco

7:12 p.m.

On Sun, Dec 3, 2023 at 10:12 AM Andrzej Krzemienski wrote:

...

Boost used to be a testing and experimentation ground before LWG. Are you saying LWG doesn't need new libraries? Or that the community doesn't need standard-library-like libraries?

I'm not saying either of things things. What I am saying is that

...

"Reinventing"? It sounds like "we are a group of people, one adventure ended, let's look for a new one". Is that what you mean?

In a sense, yes. Boost's first adventure was building world-class, peer-reviewed components destined for standardization with the accompanying field experience and bar of excellence that is Boost tradition. As authors have discovered that the bar of quality for standardization is considerably lower than that required for inclusion in the Boost library collection, Boost is no longer seen as a waypoint along the journey to standardization. So, are Beast, AEDIS, MySql and now this MQTT5 working well together? Do

...

they use similar structure, naming conventions and the design?

Yes, quite well in fact, this is precisely why I linked the Boost Servertech project! Did you have a look? All the authors and stakeholder are in the Slack C++ Language workspace working on this project together, and adjusting their respective libraries to achieve synergy (check out the #boost-servertech channel): https://github.com/anarthal/servertech-chat https://anarthal.github.io/servertech-chat/01-architecture.html https://docs.google.com/document/d/1ZQrod1crs8EaNLLqSYIRMacwR3Rv0hC5l-gfL-jO... Advantage over what?

...

An advantage over downloading individual standalone libraries from different authors, where those libraries come with no guarantee of maintenance and no peer-review. Individual libraries also do not leverage the up-front cost of installing Boost. That is to say, that if you have already integrated Boost into your project, new libraries such as Boost.Aedis or Boost.MySQL become available with no additional integration cost. Thanks

Ruben Perez

5 Dec 5 Dec

3:06 p.m.

...

So, are Beast, AEDIS, MySql and now this MQTT5 working well together? Do

...
they use similar structure, naming conventions and the design?

Yes, quite well in fact, this is precisely why I linked the Boost Servertech project! Did you have a look? All the authors and stakeholder are in the Slack C++ Language workspace working on this project together, and adjusting their respective libraries to achieve synergy (check out the #boost-servertech channel):

https://github.com/anarthal/servertech-chat https://anarthal.github.io/servertech-chat/01-architecture.html https://docs.google.com/document/d/1ZQrod1crs8EaNLLqSYIRMacwR3Rv0hC5l-gfL-jO...

As Vinnie pointed out, yes, they do. They follow Boost.Asio universal async model, which guarantees compatibility between them. ServertechChat is written using Beast, Redis and MySQL (as well as many other Boost libs). Conforming to Asio's universal model is far from easy and the first thing I check during reviews like this. You will find many "Asio based" libs out there that just use Asio and don't follow the universal model.

Rainer Deyke

2 Dec 2 Dec

2:44 p.m.

On 01.12.23 17:25, Andrzej Krzemienski via Boost wrote:

...

I thought that Boost was supposed to be an "extended Standard Library",

That hasn't been the case since Boost.Python, which has been with us since Boost 1.19.

...

containing: 1. Portability fixes 2. Implementing missing language features 3. Vocabulary types (I consider Boost.ASIO one) 4. Frequently used components 5. Ground-breaking designs and ideas

...

Async.MQTT5 doesn't fit into any of these categories. (Neither does Boost.AEDIS or Boost.MySql.)

MQTT is at least an open standard with multiple implementations, which is more than can be said for Redis or MySQL or even Python. (Alternate Python implementations exist, but Boost.Python only interfaces with the official Python implementation.) -- Rainer Deyke (rainerd@eldwood.com)

Marcelo Zimbres Silva

3 Dec 3 Dec

10:10 a.m.

On Thu, 30 Nov 2023 at 16:24, Ivica Siladic via Boost wrote:

...

Dear Boost Developers,

we are writing to introduce a new C++ Async.MQTT5 library that we believe would be a valuable addition to Boost.

<snip>

It would be great to have this functionality in Boost. I endorse it. Marcelo

Marcelo Zimbres Silva

10:11 a.m.

On Thu, 30 Nov 2023 at 16:24, Ivica Siladic via Boost wrote:

...

Dear Boost Developers,

we are writing to introduce a new C++ Async.MQTT5 library that we believe would be a valuable addition to Boost.

<snip>

It would be great to have this functionality in Boost. I endorse it. Marcelo

Ruben Perez

5 Dec 5 Dec

2:51 p.m.

On Thu, 30 Nov 2023, 16:24 Ivica Siladic via Boost, wrote:

...

Dear Boost Developers,

we are writing to introduce a new C++ Async.MQTT5 library that we believe would be a valuable addition to Boost.

I endorse reviewing this library.

357

Age (days ago)

362

Last active (days ago)

List overview

Download

15 comments

9 participants

participants (9)

Andrzej Krzemienski
Christian Mazakas
Espen Harlinn
Ivica Siladic
Marcelo Zimbres Silva
Rainer Deyke
Richard Hodges
Ruben Perez
Vinnie Falco