On Sun, May 22, 2016 at 9:49 PM, Vicente J. Botet Escriba < vicente.botet@wanadoo.fr> wrote:

Le 22/05/2016 à 14:12, Viktor Kirilov a écrit :

...

Hello!

I just released doctest - https://github.com/onqtam/doctest All the info about it can be found on github.

So do you think it can enter the boost project? How much work will it take to get it into boost except for adding boost in the title? Will it or boost benefit from that addition?

Also I've sort-of followed the Best Practice Handbook (I consider it very valuable) as much as possible - https://svn.boost.org/trac/boost/wiki/BestPracticeHandbook

Any feedback will be greatly appreciated. Thanks!

Hi,

I like the CATCH way, so I should like yours ;-)

I see that doctest doesn't depends on any Boost library. This is a good thing for a test library.

You should change the license before the review.

As Peter say, prefix all the macros by BOOST_DT_ e.g. Move the file to a include/boost folder.

Beside the markdown pages, Is there an html documentation?

Where are the test? in the example folder? Boost use to have a specific test folder with unit tets.

Vicente

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

no html documentation currently - but it can be generated from the markdown - but I'm guessing it has to look like all the rest in boost - is using boostBook/quickBook mandatory? I know there have to be some changes before submitting it for review - license, docs, tests, boost build for tests (currently only examples). I just wanted an opinion if this is a direction worth pursuing. Also I have a bunch of stuff planned in a roadmap https://github.com/onqtam/doctest/blob/master/doc/markdown/roadmap.md Do you think something from that list HAS to be implemented before submitting the library? Also reddit has exploded - https://www.reddit.com/r/cpp/comments/4kibl4/doctest_the_lightest_feature_ri...

> On May 23, 2016, at 4:14 AM, Viktor Kirilov  wrote:
> 
> On Mon, May 23, 2016 at 4:09 AM, Paul Fultz II  wrote:
> 
>> 
>>> On May 22, 2016, at 7:12 AM, Viktor Kirilov 
>> wrote:
>>> 
>>> Hello!
>>> 
>>> I just released doctest - https://github.com/onqtam/doctest
>>> All the info about it can be found on github.
>>> 
>>> So do you think it can enter the boost project? How much work will it
>> take
>>> to get it into boost except for adding boost in the title?
>>> Will it or boost benefit from that addition?
>>> 
>>> Also I've sort-of followed the Best Practice Handbook (I consider it very
>>> valuable) as much as possible -
>>> https://svn.boost.org/trac/boost/wiki/BestPracticeHandbook
>>> 
>>> Any feedback will be greatly appreciated. Thanks!
>> 
>>> everything testing-related can be removed from the binary executable by
>> defining the DOCTEST_CONFIG_DISABLE identifier
>> 
>> I’ve embedded my tests in the source code before, and I will say that its
>> a bad idea for lots of reasons: no isolation, not testing actual final
>> product, and slower tests.
>> 
>>> very small and easy to integrate - single header - less than 3k LOC in
>> the implementation translation unit and less than 1.5k LOC everywhere else
>> - extremely low footprint on compile times
>> 
>> Hmm, I don’t think its fair to say it is the lightest or small as the
>> single header is almost 3K lines of code. The Prove library[1] provides
>> testing infrastructure(and parses the test expression similar to Catch) and
>> it is only little over 400 lines of code, and it compiles pretty
>> fast(although I haven’t ran benchmarks against doctest).
>> 
>> 
> well it is not the "lightest", but the "lightest feature-rich" - there are
> some tradeoffs made

It is not that much more feature-rich than the Prove library. The Prove library is missing things like asserting for an exception and floating-point comparison(which I hope to add), but it already supports things like parametrized tests. Adding the missing features won’t make the Prove library reach 3K LOC. So I disagree with the ‘lightest’.

I'm reading this thread about doctest and I checked out the library. The first thing I noticed was macros and odd syntax. For example: TEST_CASE("testing the factorial function") { CHECK(factorial(1) == 1); CHECK(factorial(2) == 2); CHECK(factorial(10) == 3628800); } This looks like non-standard C++ and heavy on macros which turned me off to Boost.Test for the same reasons. I considered using Boost.Test for yet another library I am about to release and I gave up when I saw the baffling use of macros. People seem to be lavishing praise on these test libraries so I am asking sincerely, what am I missing here? I used my own unit test framework which is header-only, customizable, doesn't require macros in the tests, and has a test structure which is easy to generate XML reports from. Here it is: https://github.com/vinniefalco/Beast/blob/master/extras/beast/unit_test/suit... To write a test you subclass from `beast::unit_test::suite` and implement the `run()` override, and then call `expect()` for each condition that should be true. I was able to get Beast to almost 97% coverage using this simple system (see https://codecov.io/gh/vinniefalco/Beast). Here's an example of a test: https://github.com/vinniefalco/Beast/blob/master/test/core/basic_streambuf.c... Can someone please explain what I am missing? What is the benefit of macros to declare functions? What is my framework missing? Thanks On Sun, May 22, 2016 at 10:05 PM, Gavin Lambert wrote:

On 23/05/2016 00:12, Viktor Kirilov wrote:

...

I just released doctest - https://github.com/onqtam/doctest All the info about it can be found on github.

I like REQUIRE/CHECK/WARN levels.

I *really* like SUBCASEs.

Although one thing that seems a bit lacking with that syntax is support for TearDown that executes even on test failure / exception. I guess the equivalent is writing RAII guards, but these can be annoying. An execute-lambda-on-destructor class included in the framework can reduce the pain of this (or including explicit teardown syntax).

...

This allows the library to be used in more ways than any other - testscan be written directly in the production code!

I'm not convinced that this is useful beyond small utility functions (as demonstrated in the tutorial). "Real" tests often have other requirements (mocking frameworks or hand-made mocks, library substitutions [eg. in-memory db], alternate logging/output, helper methods, etc) that you'd typically want to keep out of the production code.

OTOH, it does make checking whether methods are covered by tests a bit easier if the test is right next to the method -- but [a] this is what coverage tools are for and [b] it's likely the test case wouldn't be close to the method implementation if the class requires complex setup (since it would be a subcase instead).

Once tests get sufficiently large it's easier to have them in a separate file anyway so you can toggle between the test and the code under test quickly. (Many editors are better at jumping between files rather than multiple points within a single file.)

I realise that these things are still possible with doctest, but if this wasn't a goal then I think it would simplify things or increase consistency, such as avoiding the issue with ensuring just one instantiation of the test runner / main.

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

-- Follow me on Github: https://github.com/vinniefalco

On 23/05/2016 14:29, Vinnie Falco wrote:

I'm reading this thread about doctest and I checked out the library. The first thing I noticed was macros and odd syntax. For example:

TEST_CASE("testing the factorial function") { CHECK(factorial(1) == 1); CHECK(factorial(2) == 2); CHECK(factorial(10) == 3628800); }

This looks like non-standard C++ and heavy on macros which turned me off to Boost.Test for the same reasons. I considered using Boost.Test for yet another library I am about to release and I gave up when I saw the baffling use of macros.

Generally macros are used for assertions because you want to include __LINE__ (and typically also __FILE__) in the assertion failure so that you can find *which* assertion failed (often the condition alone is not sufficiently unambiguous), and there isn't really any better way to get those AFAIK. (Sometimes it's possible to extract the failure location from a thrown exception, but this is highly compiler-dependent and non-standard -- and you're not interested in that specific point, but in one of its callers.) Using macros for the test case definition is more avoidable but is still generally done to avoid boilerplate (in this case, presumably at least auto-registration).

People seem to be lavishing praise on these test libraries so I am asking sincerely, what am I missing here? I used my own unit test framework which is header-only, customizable, doesn't require macros in the tests, and has a test structure which is easy to generate XML reports from. Here it is: https://github.com/vinniefalco/Beast/blob/master/extras/beast/unit_test/suit...

Your version seems to not capture line numbers and thus requires coming up with a unique string reason for each assertion, which is cumbersome.

To write a test you subclass from `beast::unit_test::suite` and implement the `run()` override, and then call `expect()` for each condition that should be true. I was able to get Beast to almost 97% coverage using this simple system (see https://codecov.io/gh/vinniefalco/Beast). Here's an example of a test: https://github.com/vinniefalco/Beast/blob/master/test/core/basic_streambuf.c...

This also requires listing each test twice (once in implementation, once in run()), and each suite twice (one in implementation, once in BEAST_DEFINE_TESTSUITE... which is one of those macros you didn't like, isn't it?). Requiring separate listing is "old school" test runner design; it went out of fashion because it's too error-prone (it's easy to forget to list a test case, and if you're not ensuring that tests fail before you make them pass then you might think the test passed instead of simply not being run). This design also makes it harder to run or debug individual test cases within a suite. (I don't know how easy doctest makes this, but most of the popular test runners can do this very easily, without recompiling.) You can comment out some of them but this requires a rebuild and is also error-prone since you have to remember to uncomment them again later. If it works for you, that's fine -- but these are some of the reasons why test frameworks generally avoid those things and use macros and auto-registration instead.

On Mon, May 23, 2016 at 11:57 AM, Andrzej Krzemienski wrote:

2016-05-23 5:26 GMT+02:00 Gavin Lambert :

...

On 23/05/2016 14:29, Vinnie Falco wrote:

...

I'm reading this thread about doctest and I checked out the library. The first thing I noticed was macros and odd syntax. For example:

TEST_CASE("testing the factorial function") { CHECK(factorial(1) == 1); CHECK(factorial(2) == 2); CHECK(factorial(10) == 3628800); }

This looks like non-standard C++ and heavy on macros which turned me off to Boost.Test for the same reasons. I considered using Boost.Test for yet another library I am about to release and I gave up when I saw the baffling use of macros.

Generally macros are used for assertions because you want to include __LINE__ (and typically also __FILE__) in the assertion failure so that you can find *which* assertion failed (often the condition alone is not sufficiently unambiguous), and there isn't really any better way to get those AFAIK. (Sometimes it's possible to extract the failure location from a thrown exception, but this is highly compiler-dependent and non-standard -- and you're not interested in that specific point, but in one of its callers.)

Another reason is that you can can catch an exception thrown from an expression you are testing, and report it uniformly along other test failures.

Regards, &rzej

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

The author of Catch explained 1 year ago why he uses macros here - http://accu.org/index.php/journals/2064 I chose to do the same for the same reasons - and the ones in the current mail thread.

Andrzej Krzemienski writes:

Let me remark, that Boost.Test also has this feature from Catch that it does not treat BOOST_TEST(a == b) as one expression, but instead decomposes the expression tree.

BOOST_TEST offers, much *much* more than anything Catch or any other alternatives I've seen offers. To be fair - it takes more than 400 lines of code.

The only advantage over Boost.Test seems to be the compile times, and that is an important feature, I agree.

Let's be real: how faster is it? On my 3 year old laptop single header test takes within a second (can't say for sure - it's too quick) to compile. Not to mention that you can easily build against the library if this would ever a concern. Gennadiy

On 23/05/2016 21:35, Gennadiy Rozental wrote:

Why is this called doctest? Are you testing documentation? Or doctors ;)

Presumably it's from the premise that "tests are documentation, so they should sit next to the production code" (paraphrased). Which I'm not particularly convinced by, as covered elsewhere.