Interest in a C++20 Unit Testing Framework?

older
Re: [boost] [STLInterfaces] only...

Krzysztof Jusiak

21 Nov 2019 21 Nov '19

3:23 p.m.

Hi, I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies? Github: https://github.com/boost-experimental/ut Additional links: - Try it online: https://godbolt.org/z/uVDxkW - Benchmarks: https://github.com/boost-experimental/ut#benchmarks - How it works?: https://github.com/boost-experimental/ut#how-it-works Thanks, Kris

Show replies by date

Hans Dembinski

22 Nov 22 Nov

1:36 p.m.

...

On 21. Nov 2019, at 16:23, Krzysztof Jusiak via Boost wrote:

I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

Github: https://github.com/boost-experimental/ut

It is impressive. I didn't know that catch2 takes so much computation time. Could you include Boost.Test and lightweight_test from Boost.Core in your benchmark? The small binary size and compilation time suggests that some of the tests are completely evaluated at compile-time. Is that what happens? Syntax-wise, the absence of macros is impressive, although there is a steep price to pay, I have to writing out lambdas and literals excessively, which leads to a lot of squiggly characters to type. I don't like macros, but the downsides of macros are not relevant in a test framework. Best regards, Hans

Raffi Enficiaud

8:05 p.m.

On 22.11.19 14:36, Hans Dembinski via Boost wrote:

...

...
On 21. Nov 2019, at 16:23, Krzysztof Jusiak via Boost wrote:

I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

Github: https://github.com/boost-experimental/ut

It is impressive. I didn't know that catch2 takes so much computation time. Could you include Boost.Test and lightweight_test from Boost.Core in your benchmark?

I would also be interested by that!

...

The small binary size and compilation time suggests that some of the tests are completely evaluated at compile-time. Is that what happens?

Syntax-wise, the absence of macros is impressive, although there is a steep price to pay, I have to writing out lambdas and literals excessively, which leads to a lot of squiggly characters to type. I don't like macros, but the downsides of macros are not relevant in a test framework.

I also agree with this: lack of macros is not IMO a selling point, except when it comes to the dependencies that are pulled to get something powerful enough in the preprocessor (like this is the case with boost.test and its dependency to boost.preprocess). Raffi

Vinnie Falco

8:53 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users > I was wondering whether there is any interest in exploring

...

a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

I have no interest in a library that requires C++20, especially considering that C++20 is not even official yet but also because once C++20 is released, there will be hardly any users for many years. This project seems very much like it was written "just because", to use the latest language features, rather than for pragmatic reasons. I don't see anything compelling to use it over Boost.LightweightTest for example. Thanks

Mateusz Loskot

9:28 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

(CC'ed to boost-users) On Fri, 22 Nov 2019 at 21:53, Vinnie Falco via Boost wrote:

...

On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users >

...
I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

I have no interest in a library that requires C++20, especially considering that C++20 is not even official yet but also because once C++20 is released, there will be hardly any users for many years. This project seems very much like it was written "just because", to use the latest language features, rather than for pragmatic reasons. I don't see anything compelling to use it over Boost.LightweightTest for example.

I had a look at the current documentation and examples, and I'm glad to see I'm not the only sceptical about it. My opinion is very similar to what Hans, Raffi and Vinnie explained. Macros in testing and benchmarking frameworks are kosher, in fact, they can be very helpul to organize and structure large amount of tests that in turn makes it easier to search and browse through the code. I can't imagine how any of the modern IDE's will support the string-driven syntax. As a long time maintainer of SOCI library, I'm no stranger to the syntax-first library development [1], still, I find the UT's string-based test cases approach as an interesting curiosity with very little practical value. (It falls into similar drawer as the (over)use of emoji in commit messages on GitHub :)). Boost is known as a C++ guinea pigs playground, so any library can make it in. If it does, I just hope Boost will not allow to adopt the UT as a test framework of choice of any/too many of its libraries. [1] https://www.drdobbs.com/database/a-simple-oracle-call-interface/184405930?pg... Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Krzysztof Jusiak

23 Nov 23 Nov

3:31 a.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

Thank you for your feedback. I appreciate it. Well, IMHO, there are pros and cons to any solution; I'm not going to start a flame war here about whether macros are evil or not. However, with [Boost].UT there is nothing stopping anyone from using simple macros (one-lines) to achieve other frameworks syntax The good bit about it is that it's an opt-in 'feature' as opposed to being the only available option (see example below [1]). This way more users can be satisfied and other benefits such as faster execution, quicker compilation times and other features are still available. ``` #define EXPECT(...) ::boost::ut::expect(::boost::ut::that % __VA_ARGS__) #define SUITE ::boost::ut::suite _ = [] #define TEST(name) ::boost::ut::detail::test{"test", name} = [=]() mutable SUITE { TEST("suite") { EXPECT(42 == 42); }; }; int main() { TEST("macro") { EXPECT(1 != 2); }; TEST("vector") { std::vector<int> v(5); EXPECT(5u == std::size(v)) << "fatal"; TEST("resize bigger") { v.resize(10); EXPECT(10u == std::size(v)); }; }; } ``` [1]: https://godbolt.org/z/tvy-nP -Kris On Fri, Nov 22, 2019 at 2:27 PM Mateusz Loskot via Boost-users < boost-users@lists.boost.org> wrote:

...

(CC'ed to boost-users)

On Fri, 22 Nov 2019 at 21:53, Vinnie Falco via Boost wrote:

...
On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users < boost-users@lists.boost.org>>

...
I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

I have no interest in a library that requires C++20, especially considering that C++20 is not even official yet but also because once C++20 is released, there will be hardly any users for many years. This project seems very much like it was written "just because", to use the latest language features, rather than for pragmatic reasons. I don't see anything compelling to use it over Boost.LightweightTest for example.

I had a look at the current documentation and examples, and I'm glad to see I'm not the only sceptical about it. My opinion is very similar to what Hans, Raffi and Vinnie explained.

Macros in testing and benchmarking frameworks are kosher, in fact, they can be very helpul to organize and structure large amount of tests that in turn makes it easier to search and browse through the code. I can't imagine how any of the modern IDE's will support the string-driven syntax.

As a long time maintainer of SOCI library, I'm no stranger to the syntax-first library development [1], still, I find the UT's string-based test cases approach as an interesting curiosity with very little practical value. (It falls into similar drawer as the (over)use of emoji in commit messages on GitHub :)).

Boost is known as a C++ guinea pigs playground, so any library can make it in. If it does, I just hope Boost will not allow to adopt the UT as a test framework of choice of any/too many of its libraries.

[1] https://www.drdobbs.com/database/a-simple-oracle-call-interface/184405930?pg...

Best regards, -- Mateusz Loskot, http://mateusz.loskot.net _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org https://lists.boost.org/mailman/listinfo.cgi/boost-users

Mateusz Loskot

12:19 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, 23 Nov 2019 at 04:31, Krzysztof Jusiak wrote:

...

Thank you for your feedback. I appreciate it.

In my previous post I forgot to add one more point: IMO, the library like UT should be implemented as a facade for Catch2, Boost.Test or whatever. Then, it would offer unique features with real value to users, other than the modern macro-less syntax sugar. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Bjørn Roald

12:39 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

...

On 23 Nov 2019, at 13:19, Mateusz Loskot via Boost wrote:

On Sat, 23 Nov 2019 at 04:31, Krzysztof Jusiak wrote:

...
Thank you for your feedback. I appreciate it.

In my previous post I forgot to add one more point:

IMO, the library like UT should be implemented as a facade for Catch2, Boost.Test or whatever. Then, it would offer unique features with real value to users, other than the modern macro-less syntax sugar.

I agree in principle that common interfaces would be a huge advantage, however I can not agree that an endeavor to make macro less facilities for C++ whenever we do not need macros anymore, are not worth working on. Whether the time is right, well we will see. But then we find what is missing, and that is something too. — Bjørn

Hans Dembinski

24 Nov 24 Nov

4:55 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

...

On 23. Nov 2019, at 04:31, Krzysztof Jusiak via Boost wrote:

...

However, with [Boost].UT there is nothing stopping anyone from using simple macros (one-lines) to achieve other frameworks syntax The good bit about it is that it's an opt-in 'feature' as opposed to being the only available option (see example below [1]).

But then you have split in the user base, some will use the macros others will use the macro-free syntax, and both will have difficulty in understanding the code of other.

Vinnie Falco

5:50 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sun, Nov 24, 2019 at 9:49 AM Hans Dembinski via Boost wrote:

...

both will have difficulty in understanding the code of other.

BOOST_TEST(b) What could this possibly mean? /me scratches head Regards

Krzysztof Jusiak

6:26 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

I see your point, but I'm not advocating here for providing multiple ways of doing the same thing as I agree it might be confusing although I'd also argue that it's already a case with boost.test for example (single header/static/shared library or BOOST_TEST, BOOST_CHECK, BOOST_CHECK_EQ accomplish pretty much the same things but there are also a trade-offs here so it depends) Also, having a possibility for users to do so is valuable IMHO but I wouldn't expose any macros from the library. On Sun, Nov 24, 2019 at 9:55 AM Hans Dembinski wrote:

...

...
On 23. Nov 2019, at 04:31, Krzysztof Jusiak via Boost < boost@lists.boost.org> wrote:

...
However, with [Boost].UT there is nothing stopping anyone from using simple macros (one-lines) to achieve other frameworks syntax The good bit about it is that it's an opt-in 'feature' as opposed to being the only available option (see example below [1]).

But then you have split in the user base, some will use the macros others will use the macro-free syntax, and both will have difficulty in understanding the code of other.

Raffi Enficiaud

7:15 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On 24.11.19 19:26, Krzysztof Jusiak via Boost wrote:

...

I see your point, but I'm not advocating here for providing multiple ways of doing the same thing as I agree it might be confusing although I'd also argue that it's already a case with boost.test for example (single header/static/shared library or BOOST_TEST, BOOST_CHECK, BOOST_CHECK_EQ accomplish pretty much the same things but there are also a trade-offs here so it depends)

For info, BOOST_TEST does everything (and more): https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/testing_... says "BOOST_TEST: universal and general purpose assertions" The other macros are mostly kept for C++03 compilers and compatibility. For the header/static/shared, this is beyond C++ itself, and IMO this is needed when you want to address various code bases, although it creates a lot of confusion on the users' side. Don't get me wrong, I have nothing against or in favor of the UT library. Since we are talking about "standard" way of testing C++ code, here are some properties I believe are common or established [*]: * reporting in various formats, actionable runtime, including the ones that are understood by CI * floating point comparison * xUnit type of testing (fixture, suites, cases) * "robust" to various type of exceptions All the rest is bonus. Raffi [*] I have not checked what UT provides, I am a user of Boost.Test and GoogleTest

Bjørn Roald

23 Nov 23 Nov

12:56 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

...

On 22 Nov 2019, at 21:53, Vinnie Falco via Boost wrote:

On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users > I was wondering whether there is any interest in exploring

...
a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

I have no interest in a library that requires C++20, especially considering that C++20 is not even official yet but also because once C++20 is released, there will be hardly any users for many years. This project seems very much like it was written "just because", to use the latest language features, rather than for pragmatic reasons. I don't see anything compelling to use it over Boost.LightweightTest for example.

Is your lack of interest mostly based on its lack of potential utility for your projects now or your lack of interest for such tools in the future? What are the chances new libraries depending heavily on macros in their interfaces will ever make in into the standard library? If they will not I don’t think it is fair to say this is something that look like someone just want to play with the newest language features, and that it has no other merit. I agree there may be elements of the library that aligns with your assessment, but I suspect that is for good reasons where options where not available other than macros. If that is the case, we have learned something new, haven’t we? — Bjørn

Vinnie Falco

2:05 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, Nov 23, 2019 at 4:56 AM Bjørn Roald via Boost wrote:

...

Is your lack of interest mostly based on its lack of potential utility for your projects now or your lack of interest for such tools in the future?

I don't see a compelling case for a unit-test library to be standardized. And I see nothing wrong with macros. They are not going away, ever. Thanks

Richard Hodges

2:13 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

...

I don't see a compelling case for a unit-test library to be standardized.

I respectfully disagree here. Standardisation aids adoption, learning and portability. My view is that c++ would be more popular if the standard was wider in scope. The arguments I hear against using c++ in new projects include “you need arcane knowledge to actually achieve anything”. It is for this reason that I regard boost beast as the most pivotal library to bless the c++ community since the boost project was started. And I see nothing wrong with macros. Except that they pollute the global namespace and contribute to unintuitive bugs? They are not going

...

away, ever.

We can hope and work in that direction.

...

Thanks

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

-- Richard Hodges hodges.r@gmail.com office: +442032898513 home: +376841522 mobile: +376380212

Vinnie Falco

2:15 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, Nov 23, 2019 at 6:14 AM Richard Hodges via Boost wrote:

...

Standardisation aids adoption, learning and portability.

What does being in the standard offer that cannot be achieved as an independent library? Thanks

Bjørn Roald

2:22 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

...

On 23 Nov 2019, at 15:15, Vinnie Falco via Boost wrote:

On Sat, Nov 23, 2019 at 6:14 AM Richard Hodges via Boost wrote:

...
Standardisation aids adoption, learning and portability.

What does being in the standard offer that cannot be achieved as an independent library?

When adapted into your project, not much. But as far as getting there, a lot. In some projects, a lot more than you may believe, even for a unit test tool. — Bjørn

Vinnie Falco

2:45 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, Nov 23, 2019 at 6:22 AM Bjørn Roald via Boost wrote:

...

When adapted into your project, not much. But as far as getting there, a lot. In some projects, a lot more than you may believe, even for a unit test tool.

This doesn't answer the question. Here's an example of how the question might be answered for a popular Boost library, Boost.Asio: "C++ benefits from standardized networking, because having common vocabulary types and named requirements allows libraries written against standardized networking components to interoperate." The same cannot be said for unit test libraries, since those types don't appear in public interfaces. So again what is the benefit of standardization? Thanks

Krzysztof Jusiak

3:17 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

...

So again what is the benefit of standardization?

IMHO, there is always a benefit of standardization best/common practices because * It lowers the entry-level to the language (no need for third-party libraries) * It improves the education aspect (one standard way of doing it) * It makes the language more coherent/stable (consistent design with other features, stable API) * It makes the feature a first class citizen (shows that the community cares about this aspect of the language) to name a few... On Sat, Nov 23, 2019 at 7:45 AM Vinnie Falco via Boost < boost@lists.boost.org> wrote:

...

On Sat, Nov 23, 2019 at 6:22 AM Bjørn Roald via Boost wrote:

...
When adapted into your project, not much. But as far as getting there, a lot. In some projects, a lot more than you may believe, even for a unit test tool.

This doesn't answer the question.

Here's an example of how the question might be answered for a popular Boost library, Boost.Asio:

"C++ benefits from standardized networking, because having common vocabulary types and named requirements allows libraries written against standardized networking components to interoperate."

The same cannot be said for unit test libraries, since those types don't appear in public interfaces. So again what is the benefit of standardization?

Thanks

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Vinnie Falco

3:33 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, Nov 23, 2019 at 7:17 AM Krzysztof Jusiak wrote:

...

* It lowers the entry-level to the language (no need for third-party libraries)

It should be obvious to everyone that external libraries cannot, and should not be an impediment to adoption. Similarly, adding every possible library to the standard is obviously not sustainable. Yes there's a problem with packaging, but the solution is not to propose everything for the standard. It is to solve the packaging problem (perhaps that could be the focus of your new efforts?) "Because being in the standard makes it easier to include" is not a sufficient justification for being in the standard.

...

* It improves the education aspect (one standard way of doing it)

One of the strengths of C++ is that it does not impose any particular models of computation or paradigms on the user. This is important for many reasons, performance being one but also to allow problems to be solved in different ways, with each solution ideally suited to meet the needs of a particular use-case. "Because it becomes easier to teach" is not a sufficient justification for being in the standard.

...

* It makes the feature a first class citizen (shows that the community cares about this aspect of the language)

"Showing that the community cares" is not a sufficient justification for being in the standard. The C++ Standard is effectively a legal document which mandates a set of requirements that must be fulfilled for a vendor to claim that their product is "C++." Everything added to the standard increases the burden for all implementers. It is very easy for a random Internet person to propose a new library for the C++ standard, because they do not bear the cost of implementation on all platforms and toolchains. They don't even bear the cost of the initial proposed wording (I see no wording from you, and a dearth of documentation in general). And they don't bear the cost of refining the wording to resolve conflicts before getting into the draft. There must be an overwhelming preponderance of supporting evidence that anything added to the standard is worth far more than the cost. I don't see that in your library. Thanks

Krzysztof Jusiak

4:14 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

My personal take is that C++ strength comes from standardizing zero-overhead abstractions and not the underlying implementation which allows for the best possible implementation depending on a variety of factors. I also believe that having basic testing primitives in the standard would benefit C++ (as stated before) but I understand that opinions will vary here and I'm not going to argue that (there are pros and cons both). I'm also definitely not proposing [Boost].UT for standardization here; just checking whether there is an interest in the library, that's it. On Sat, Nov 23, 2019 at 8:33 AM Vinnie Falco wrote:

...

...
* It lowers the entry-level to the language (no need for third-party

On Sat, Nov 23, 2019 at 7:17 AM Krzysztof Jusiak wrote: libraries)

It should be obvious to everyone that external libraries cannot, and should not be an impediment to adoption. Similarly, adding every possible library to the standard is obviously not sustainable. Yes there's a problem with packaging, but the solution is not to propose everything for the standard. It is to solve the packaging problem (perhaps that could be the focus of your new efforts?)

"Because being in the standard makes it easier to include" is not a sufficient justification for being in the standard.

...
* It improves the education aspect (one standard way of doing it)

One of the strengths of C++ is that it does not impose any particular models of computation or paradigms on the user. This is important for many reasons, performance being one but also to allow problems to be solved in different ways, with each solution ideally suited to meet the needs of a particular use-case.

"Because it becomes easier to teach" is not a sufficient justification for being in the standard.

...
* It makes the feature a first class citizen (shows that the community cares about this aspect of the language)

"Showing that the community cares" is not a sufficient justification for being in the standard.

The C++ Standard is effectively a legal document which mandates a set of requirements that must be fulfilled for a vendor to claim that their product is "C++." Everything added to the standard increases the burden for all implementers. It is very easy for a random Internet person to propose a new library for the C++ standard, because they do not bear the cost of implementation on all platforms and toolchains. They don't even bear the cost of the initial proposed wording (I see no wording from you, and a dearth of documentation in general). And they don't bear the cost of refining the wording to resolve conflicts before getting into the draft.

There must be an overwhelming preponderance of supporting evidence that anything added to the standard is worth far more than the cost. I don't see that in your library.

Thanks

Mateusz Loskot

3:42 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, 23 Nov 2019 at 16:17, Krzysztof Jusiak via Boost wrote:

...

...
So again what is the benefit of standardization?

IMHO, there is always a benefit of standardization best/common practices because

Good, then the documentation, in the motivation section, could/should answer: What are the best practices the UT captures? in comparison to popular existing frameworks, which do not necessarily use the best practices. My take on the main points listed in the https://github.com/boost-experimental/ut/blob/master/README.md#overview, - no dependencies - it's a convenience, not best practice - single header - header-only libraries are sometimes it is a convenience, sometimes a necessity, sometimes a matter of personal preference or choice, but header-only is not a best practice. - macro-free - generally, this is best practice; specifically, it depends. As Vinnie pointed out, macros (polluting global namespace) in test frameworks are not a troublemaker, so in case of UT this is a weak argument. - easy to use - yes, this is a universal best practice. - minimal API - well, most test framework I've seen or used allow users to get the job done using 2-3 macros e.g. using Catch2 you need to learn just two. 2-5 macros is minimal API. - fast to compile - Can this be called a best practice? Well, it certainly is a nirvana to aim for but in real case, it depends. Boost.MP11 compiles faster than Boost.MPL but some projects have no choice than to use MPL. If the UT aims for an ideal of standardization proposable library, then it (documentation) needs to make the best practices more vivid. I do like your library. It is a very interesting experiment, a very good piece of code to read and learn from about C++20 features. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Krzysztof Jusiak

3:59 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

I wasn't even trying to imply that [Boost].UT should go to the standard (it's not even a boost library ;) but, as stated, I believe there are benefits of having things in the standard and I, personally, think that C++ would benefit from having some basic testing primitives but, as always there are pros and cons to literally anything. On Sat, Nov 23, 2019 at 8:43 AM Mateusz Loskot via Boost < boost@lists.boost.org> wrote:

...

On Sat, 23 Nov 2019 at 16:17, Krzysztof Jusiak via Boost wrote:

...
...
So again what is the benefit of standardization?

IMHO, there is always a benefit of standardization best/common practices because

Good, then the documentation, in the motivation section, could/should answer: What are the best practices the UT captures? in comparison to popular existing frameworks, which do not necessarily use the best practices.

My take on the main points listed in the https://github.com/boost-experimental/ut/blob/master/README.md#overview,

- no dependencies - it's a convenience, not best practice

- single header - header-only libraries are sometimes it is a convenience, sometimes a necessity, sometimes a matter of personal preference or choice, but header-only is not a best practice.

- macro-free - generally, this is best practice; specifically, it depends. As Vinnie pointed out, macros (polluting global namespace) in test frameworks are not a troublemaker, so in case of UT this is a weak argument.

- easy to use - yes, this is a universal best practice.

- minimal API - well, most test framework I've seen or used allow users to get the job done using 2-3 macros e.g. using Catch2 you need to learn just two. 2-5 macros is minimal API.

- fast to compile - Can this be called a best practice? Well, it certainly is a nirvana to aim for but in real case, it depends. Boost.MP11 compiles faster than Boost.MPL but some projects have no choice than to use MPL.

If the UT aims for an ideal of standardization proposable library, then it (documentation) needs to make the best practices more vivid.

I do like your library. It is a very interesting experiment, a very good piece of code to read and learn from about C++20 features.

Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Mateusz Loskot

4:09 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, 23 Nov 2019 at 16:59, Krzysztof Jusiak wrote:

...

I wasn't even trying to imply that [Boost].UT should go to the standard (it's not even a boost library ;) but, as stated, I believe there are benefits of having things in the standard and I, personally, think that C++ would benefit from having some basic testing primitives but, as always there are pros and cons to literally anything.

Yes, you were not. Since you touched on best practices as one of goals/roles of the standardization, I just took it as an opportunity to briefly review what the current UT docs present in terms of the possible best practices. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Vinnie Falco

4:16 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, Nov 23, 2019 at 8:00 AM Krzysztof Jusiak via Boost wrote:

...

I wasn't even trying to imply that [Boost].UT should go to the standard

Yeah, you weren't. That was Bjørn Roald. Although you did claim some benefits to standardization, which I replied to. Thanks

Bjørn Roald

4:34 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

...

On 23 Nov 2019, at 17:16, Vinnie Falco via Boost wrote:

On Sat, Nov 23, 2019 at 8:00 AM Krzysztof Jusiak via Boost wrote:

...
I wasn't even trying to imply that [Boost].UT should go to the standard

Yeah, you weren't. That was Bjørn Roald. Although you did claim some benefits to standardization, which I replied to.

Well, I don’t think I was proposing it for standardization. Not that I do not think it may be worth it down the road. But for now it is most likely far from being ready for that. What I am proposing is that it may be worth exploring this type of library for possible future standardization, and that is why I think it may be worth considering for Boost. I will see if I can spend some more time with it, do some testing, and see if it affect my opinion. I agree with you Vinnie that it does not provide much functionally over alternatives. I think you only see it as syntactic sugar what I see as a potential path to something new in standard C++. Similar technicues may be useful not only unit test, but also for diagnostic logging or other tools. — Bjørn

Mateusz Loskot

10 Dec 10 Dec

1:58 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Sat, 23 Nov 2019 at 16:17, Krzysztof Jusiak via Boost wrote:

...

...
So again what is the benefit of standardization?

IMHO, there is always a benefit of standardization best/common practices because

FYI, Reddit has just asked "Should the standard library include a unit test framework? " https://old.reddit.com/r/cpp/comments/e7vcpb/should_the_standard_library_inc... with some interesting POVs, including from those who work on ISO C++ itself, like Bryce's https://old.reddit.com/r/cpp/comments/e7vcpb/should_the_standard_library_inc... Best regards, -- Mateusz Loskot, http://mateusz.loskot.net

Gavin Lambert

24 Nov 24 Nov

11:12 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On 24/11/2019 03:45, Vinnie Falco wrote:

...

The same cannot be said for unit test libraries, since those types don't appear in public interfaces. So again what is the benefit of standardization?

Some possible benefits to standardising a unit test framework: 1. The unit tests for a standard library can be published in the standard as well (using the standard unit test library), thereby acting as additional documentation on intended behaviour and a way to verify whether a particular implementation is conforming. 2. Having a standard vocabulary for unit tests that are likely to be portable to more platforms and more universally understood by all developers (once familiar with the new standard), rather than seemingly every project using a different framework. Having said that, I'm not entirely convinced that it is practical or even desirable to do so, since there's still quite a lot of religious debates about testing, such as fakes/mocks, expect vs assert, catch2 nested tests vs suites, AAA mini-tests vs. kitchen sink tests, etc. And (especially when using mocks) it can be a little too easy to fall into the trap of asserting the specific implementation instead of the general behaviour, which is expressly what the standard wants to avoid.

Krzysztof Jusiak

27 Nov 27 Nov

9:09 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

FYI, [Boost].UT (since v1.1.2) has relaxed the standard requirement to C++17 (although some limitations apply depending on the standard/compiler combination). In general, source_location is available since C++20, however, the builtin functionality powering it (__builtin_FILE(), __builtin_LINE()) is available since GCC-9/Clang-9. Therefore, now: * [Boost].UT with Clang/GCC >= 9 and with either C++17 or C++20 works as expected (although with C++17 it uses compiler builtin designed for C++20 source_location) * [Boost].UT with Clang/GCC < 9 and with either C++17 or C++20 still compiles/`works`, however, the file/line in assertion won't be propagated Also, MSVC-2019 doesn't support source_location yet neither with /std:C++20 nor with /std:C++latest, hence, the file/line is also not propagated yet. Example here -> https://godbolt.org/z/XNgMdN -Kris On Fri, Nov 22, 2019 at 1:53 PM Vinnie Falco wrote:

...

On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users > I was wondering whether there is any interest in exploring

...
a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

I have no interest in a library that requires C++20, especially considering that C++20 is not even official yet but also because once C++20 is released, there will be hardly any users for many years. This project seems very much like it was written "just because", to use the latest language features, rather than for pragmatic reasons. I don't see anything compelling to use it over Boost.LightweightTest for example.

Thanks

Bjørn Roald

23 Nov 23 Nov

12:33 p.m.

...

On 21 Nov 2019, at 16:23, Krzysztof Jusiak via Boost wrote:

Hi,

I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

From a quick look I think this is an interesting project with some impressive results. I can probably not use it any time soon in its current form and shape for other than some test projects. That may change sooner than I think, as newer compilers supporting C++20 may become generally used where I work and all testing of our projects can assume C++20 is available. For many existing projects this may take some time, or never be feasible, based on their code base, target platforms, as well as time and funding to do migrations. Even new projects may be required to support older C++ dialects. Hence, one downside of this sort of project is that the potential number of users are not large from the get-go. For many potential users there are alternatives that are more accessible and hence more feasible choices. As we can see in this thread, some seems to declare their dis-interest in the library on this basis. I think that is unfortunate and wanted to voice my view of support and interest. If there is a home for a project like this, why is that not in boost? Even though there may be several iterations with refinements before such a project reach its stable interfaces and concepts. To get there, and then possibly on that basis provide interfaces for or influence standardizing of unit testing, we need someone to explore this space. There is not going to be a MACRO based standard unit test library. Are there anyone here that is disputing that? If we agree on that, then in my view it follows that Boost is the right home for this sort of project. Reading the examples, there are parts of how test cases are defined that I at best need time to get used too. But worst is that I assume this is so as there are not limitless options given that one are not using macros, even in C++20. So there may be valid arguments that the language is not ready for this yet even with source_location in the standard. Compile time reflection did not make it into C++20, and I can envision it might have provided better alternatives, but I am not sure nor an expert. I am also sure there are other wrinkles in this library that need to be ironed out. This may happen before an eventual acceptance into boost, or later if included while a Boost library evolving toward a good model for standardized unit test facilities for C++. I suspect some I hope there are sufficient support for this project and similar projects in Boost. I think such projects will have better chances of success than outside of Boost. In addition, Boost libraries are accessible to developers in many companies where other libraries are generally not. If not accepted into Boost, then I am sure we soon will see this or similar libraries evolve or be invented outside of Boost. Given that scenario, I would be left to wonder what the Boost mission is about. At least part of it I hope is still to push the boundaries of what can be done in C++ and what C++ will become. — Bjørn

Vinnie Falco

24 Nov 24 Nov

6:23 p.m.

New subject: [Boost-users] Interest in a C++20 Unit Testing Framework?

On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users wrote:

...

I was wondering whether there is any interest in exploring a C++20 single header/single module, macro-free Unit Testing Framework with no dependencies?

Apologies if my previous comments were discouraging. There is benefit to "exploring" this library, in that it is useful to see what a re-imagining of a unit test interface could look like if it used C++20 constructs instead of macros. Using a user defined string literal to name the test case is innovative as are some of the other features. My comments are narrowly focused only on the practical use of the library in a general sense. Regards

1813

Age (days ago)

1832

Last active (days ago)

List overview

Download

30 comments

8 participants

participants (8)

Bjørn Roald
Gavin Lambert
Hans Dembinski
Krzysztof Jusiak
Mateusz Loskot
Raffi Enficiaud
Richard Hodges
Vinnie Falco