On Sep 10, 2013, at 21:12, Gennadiy Rozental wrote:

Jared Grubb writes:

...

I recently read "Modern C++ Programming with Test-Driven Development",

... All these books... trying to reinvent the wheel.

...

and I really enjoyed this book and highly recommend it. One of the styles I saw for the first time was "hamcrest"-style assertions. ("Hamcrest" is an anagram of "matchers"; also see http://en.wikipedia.org/wiki/Hamcrest.)

For a quick idea, imagine there is a BOOST_CHECK_THAT macro that looks like this: (excuse my choice of names here, we can pick better ones if people like the idea)

BOOST_CHECK_THAT( str, IsEqualTo(expStr) ); BOOST_CHECK_THAT( str, HasSubstring(needleStr) ); BOOST_CHECK_THAT( str, Not(HasSubstring(badStr)) ); BOOST_CHECK_THAT( vec, Contains(42) );

Comparing them to current boost::test assertions, usually the two are directly equivalent, and it's just a matter of style. At first, I was unconvinced of a need for this. However, there are a few scenarios where the hamcrest-style actually are really helpful:

BOOST_CHECK_THAT(vec, Any(Contains(42), Contains(-42)) ); BOOST_CHECK_THAT(str, All(StartsWith("abc"), HasSubstring("lmnop"), EndsWith("xyz")) ); BOOST_CHECK_THAT(count, All(GE(0), LT(10)) ); BOOST_CHECK_THAT(vec, HasSubsequence(otherVec) ); BOOST_CHECK_THAT(vec, Sorted(IsEqualTo({1, 2, 3})) ); BOOST_CHECK_THAT(optionalStr, Any(IsEqualTo(boost::none), Deref(IsEqualTo("str")));

Hi Jared,

Let me start by saying that you might have something here, but I am not convinced yet. I have number of comments here:

1. Stylistic All these IsEqualTo, HasSubstring etc has a strong Java stench. I personally do not see a place for something like this in proper C++.

Jeff Langr, the author of that book, is indeed a Java guy ... your instinct is right on. :) I'm with you on this. I wasnt convinced by them at first either; I personally really dislike Java style.

2. Why exactly you are giving preference to first argument over all other predicate arguments? Will your predicate report them all? Why the asymmetry then?

Yes, the diagnostics can take advantage of that; for example, "got 42, expected 43" instead of "42 != 43".

3. Boost.Test already have this macro. It is called BOOST_CHECK_PREDICATE

BOOST_CHECK_PREDICATE( P, (a)(b)(c) );

Very nice symmetry and most of your your other points covered (stackability etc)

I hadnt seen that one; I tried it, but it does require extra parens that I find distracting: BOOST_CHECK_PREDICATE(vec, (Contains(42)));

Moreover we have a framework for writing custom predicates. This should be a function which returns test_tools::assertion_result. For example:

boost::test_tools::assertion_result compare_lists( std::list<int> const& l1, std::list<int> const& l2 ) { if( l1.size() != l2.size() ) { boost::test_tools::predicate_result res( false );

res.message() << "Different sizes [" << l1.size() << "!=" << l2.size() << "]";

return res; }

return true; }

Yes, these new predicates would be implemented like this.

4. The new Boost.Test drops most of the current macros in a favor of a single "super" macro BOOST_TEST:

BOOST_TEST( a == b ) BOOST_TEST( a != b ) BOOST_TEST( a+1 <= b )

This one is based on brilliant idea from Kevlin Henney (pretty much once of the few rare new ideas that came out recently in testing domain). This handles many of your cases already with proper syntax.

I'm looking forward to these! I think BOOST_TEST covers 80-90% of the assertions I would use. Even if hamcrest was available, I would still expect that people use BOOST_TEST(a==b) over BOOST_CHECK_THAT(a, Eq(b)). But, the "Eq" check is still useful in composed tests ("None(Eq,Eq)"), which is the only reason I think it should be included. However, there are a class of assertions that get a bit ugly in the BOOST_TEST form: * Contains(42) vs vec.find(42) != vec.end() * Any(Contains(42), Contains(43)) vs. ( vec.find(42) != vec.end() || vec.find(43) != vec.end()) Using a higher level concept, rather than raw STL algorithm, also lets the failure messages be a bit more coherent. Compare: * the error message that prints the inequality of "str.find("abc")" and "string::npos" (something like "10 != -1") * the error message "str does not contain "abc"; str was "xyz"" The partially-bound nature of these assertions ( where _THAT( a, A(x) ) => "A(x)(a)" ) is useful because it means the value-under-test can be written once and, even more important, the value's expression is evaluated only once. This lets you pop values from a queue and run two checks without having to save the value first to a local variable; this can improve the readability of a unit test (and it can hurt too if misused).

5. There way to many custom user specific predicates out there. We can try all we want, but all we end up doing is just covering needs of rather small subset of users. There should be line we draw somewhere: what is expected to be supplied by the library and what is specific to the user's domain.

That being said...

If you believe you can present a set of some generic reusable predicates, which can be used with BOOST_TEST_PREDICATE and implemented based on assertion_result, I think we can discuss if there is a place for it inside of UTF. Keep in mind "do not pay for what do not use" principle. You can't have one big header with 100 dependencies.

Agreed. I think a massive header full of a hamcrest corresponding to each STL algorithm is overkill, but I think that a handful of them could make testing some (10%) assertions easier. The hamcrest header(s) would have to be explicitly included (not included in unit_test.hpp) and users would probably want a "using namespace" to bring these into global lookup. Jared

Jared Grubb writes:

...

3. Boost.Test already have this macro. It is called BOOST_CHECK_PREDICATE

BOOST_CHECK_PREDICATE( P, (a)(b)(c) );

Very nice symmetry and most of your your other points covered (stackability etc)

I hadnt seen that one; I tried it, but it does require extra parens that I find distracting:

I find your asymmetry more distructing

BOOST_CHECK_PREDICATE(vec, (Contains(42)));

This obviously is wrong syntax.

However, there are a class of assertions that get a bit ugly in the BOOST_TEST form: * Contains(42) vs vec.find(42) != vec.end() * Any(Contains(42), Contains(43)) vs. ( vec.find(42) != vec.end() || vec.find(43) != vec.end())

I would use || and && and ! instead of Any, All, Not: BOOST_CHECK_PREDICATE( contains(42) || contains(65), (container) ); BOOST_CHECK_PREDICATE( contains(5) && !contains(6), (container) );

Using a higher level concept, rather than raw STL algorithm, also lets the failure messages be a bit more coherent.

There are 3 issues: * We probably to do not want to replicate whole STL with "better error message" predicates operating on containers instead of iterators * What if I do want to use iterators? * I wonder how will you combine generically messages from 3 different predicates in And predicate? We failed because "error msg 1" and "err msg 2" and "err msg 3". This can become ugly very fast.

The partially-bound nature of these assertions ( where _THAT( a, A(x) ) => "A(x)(a)" ) is useful because it means the value-under-test can be written once and, even more important, the value's expression is evaluated only once.

This is also true in BOOST_CHECK_PREDICATE case (both parts). I do not see any single reason to use this asymmetric syntax. Another complication to keep in mind is that you need to make all your predicate "silently" bindable (here silently means an absence of bind invocation. In other words both these should be valid: BOOST_CHECK_PREDICATE( contains(1), (container) ); BOOST_CHECK_PREDICATE( contains, (container, 1) ); BOOST_CHECK_PREDICATE( within_range(1,10), (value) ); BOOST_CHECK_PREDICATE( !within_range, (value,1,10) ); Gennadiy

On 9/16/2013 12:45 PM, Quoth Jared Grubb:

The && operator actually isnt all that bad, since you would use short-circuit evaluation. So, "A && B && C" would result in one of "A failed", "A passed, but B failed", or "A and B passed, but C failed".

Short-circuit evaluation typically only happens for primitive types (ie. bool). When using custom types with overloaded operators the compiler will evaluate all arguments in a non-short-circuit manner. (Even if one of them is bool -- "false && expr()" will still result in evaluating the expression and calling the && operator when the expression returns a non-primitive type that implements operator&&.) While it would be possible to print the messages as if short-circuit evaluation had been done, this would actually be discarding potentially useful information (eg. when A failed but B and C passed it suggests a different underlying fault than if all three failed). So I would suggest being verbose all the time anyway.

Assertion failed. Expected all of the following to pass: * contains(42) - passed * any of the following conditions: * contains(43) - failed * contains(44) - failed

I like this sort of syntax, except that for symmetry between the && and || operations I would expect it to look more like this: Assertion failed. * all of the following conditions: * contains(42) - passed * any of the following conditions: * contains(43) - failed * contains(44) - failed The idea being that the && and || operators would return something that would add a sibling message to a previous matching operator (for compactness of a && b && c && d etc) but would add a nested message to a different operator.

On 9/17/2013 4:20 AM, Quoth Jared Grubb:

Short-circuit is still possible because the hamcrest-assertions are partially bound. So: BOOST_CHECK_PREDICATE( A && B, (value)); becomes like "(A && B)(value)". The wrapping object around "A && B" would hold A and B as members andwould forward "value" according to short-circuit evaluation rules.

While I agree that you could do that, my point was that I'm not convinced that there would be any benefit in doing so. It seems to me that it would be more complex internally to do that, and ultimately more useful to display all the failing sub-conditions anyway (as I said before, "false && false && false" might indicate a different failure cause than "false && true && false", but short-circuiting would hide this distinction). Short-circuiting might be beneficial if you were doing a test like "p && p->Something()" but that doesn't seem like it fits into the BOOST_CHECK_PREDICATE syntax anyway. Unless you're going to make predicates such as "contains" (and all others) auto-dereference pointers, the final expression would have to be a bare object or reference anyway (not a pointer), so any predicate would fail if it were null even with short-circuiting. Or you'd have to define modifiers or something, eg: BOOST_CHECK_PREDICATE(!is_nullptr() && deref(contains(42) || contains(43)), (ptr_to_container)); (where "deref" will do "*x" before passing it to the underlying predicate). Which I'm not convinced is any prettier than the alternative of just having two separate checks. And even in this case you could handle non-short-circuiting gracefully -- deref could simply report failure without calling its internal sub-predicate if it gets passed a null pointer. This would even let you omit the leading null check.

On Dec 29, 2013, at 1:04, Richard wrote:

[Please do not mail me a copy of your followup]

boost@lists.boost.org spake the secret code thusly:

...

I recently read "Modern C++ Programming with Test-Driven Development", and I really enjoyed this book and highly recommend it.

I didn't know about this book until you posted this thread. I bought it and read it and loved it!

Yes, it's a terrific book!

...

One of the styles I saw for the first time was "hamcrest"-style assertions. ("Hamcrest" is an anagram of "matchers"; also see http://en.wikipedia.org/wiki/Hamcrest.)

This is something that I don't think really buys us much in C++. The reason I say this is because in C++ we can take a macro argument and turn it into a string in order to produce a diagnostic message for the assert -- which Boost.Test does already.

In Java, you have no preprocessor and no way to do this, so Hamcrest style matchers are the way that they get the extra detail into the assertion messages.

Combine this ability of the preprocessor to take arbitrary program source text and turn it into a string and introduce a set of useful predicates for things like checking if a string starts with a prefix and you find the need for Hamcrest style matchers pretty much disappears.

Without the ability to turn source text into a string, the case for Hamcrest style matchers would increase dramatically. One area where I see Hamcrest style matchers being useful is in configuring mock objects. Having used jMock for Java and Turtle for Boost.Test/C++ I think both of them read fairly similar.

I put together a version of the Hamcrest matchers, and the only ones that I've ended up using are: * SameSequence -- I find the "equal collections" macro to be awkward (mainly due to the use of iterators) * SameSequenceAnyOrder -- when the order of the result is not known (or relevant); the custom error strings are very helpful here * FutureReady/FutureNotReady -- shorthand for "BOOST_CHECK_EQUAL(future.wait_for(...), std::future_status::ready/timeout)" But, it is very Java-esque, so I understand no one else was very excited about adding these. But, I figured I'd ask :) Jared

-- "The Direct3D Graphics Pipeline" free book http://tinyurl.com/d3d-pipeline The Computer Graphics Museum http://computergraphicsmuseum.org The Terminals Wiki http://terminals.classiccmp.org Legalize Adulthood! (my blog) http://legalizeadulthood.wordpress.com

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost