Hello Raffi,

...

BOOST_AUTO_TEST_SUITE(....

BOOST_DATA_TEST_CASE( declaration,         dataset.input()      ^ dataset.expect()      ^ dataset.test_case_name(),      code, expect_AST, test_case_name) {      using attribute_type = ast::declaration;      auto const parser = parser::declaration;

     x3_test::testing_parser<attribute_type> parse;      auto [parse_ok, parsed_AST] = parse(code, parser);

     BOOST_TEST(parse_ok);      BOOST_REQUIRE_MESSAGE(parse_ok,          x3_test::report_diagnostic(test_case_name, code, parsed_AST)      );

     BOOST_TEST(parsed_AST == expect_AST, btt::per_element());      BOOST_REQUIRE_MESSAGE(x3_test::current_test_passing(),          x3_test::report_diagnostic(test_case_name, code, parsed_AST)      ); }

BOOST_AUTO_TEST_SUITE_END()

After some testing, the error count is twice the real (obviously here) but it's all above a work arround.

It seems, that BOOST_REQUIRE_MESSAGE evaluates always the message predicate (shown by using --log_level=all) nevertheless the test case successed or failed. If failed it is streamed to the console.

That sounds odd to me. Would it be possible to check if the messages get evaluated in case you do not pass --log_level=all ?

I reduced it and found the original problem. Checking parse_ok twice results into two messages, this explain what I faced (and misinterpreted in the last mail). The reason for introducing this was the intention of: - check on parse_ok success - if not, generate a diagnostic message - all further checks can be canceled - compare the result per_element with reference/gold data - if failed, generate a diagnostic message - note, report_diagnostic saves the parsed_AST for later inspection The problem may rise due to my limited understanding about the macros arround.

What about :

1- using test-case fixtures for writing things at the end of the test 2- registering an observer for getting the failure signal?

The observer is (kind of) described here: https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/fra...

You may install and remove it globally by using a global fixture:

https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_refe...

Once registered, it will capture the events that you want, especially the failure cases.

From the observer, you can get the current test cases with this

https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/fra...

I will check/evaluate it next time, it simplifies diagnostic since the parsed_AST can be written by the observer Thank you, Olaf

On 23.07.18 19:28, Olaf Peter via Boost-users wrote:

...

...

How to get this?

What about :

1- using test-case fixtures for writing things at the end of the test 2- registering an observer for getting the failure signal?

The observer is (kind of) described here: https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/fra...

You may install and remove it globally by using a global fixture:

https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_refe...

Once registered, it will capture the events that you want, especially the failure cases.

From the observer, you can get the current test cases with this

https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/fra...

OK, my test_observer_fixture with overloads is ready:

class test_observer_fixture {     struct test_observer : public utf::test_observer     {         virtual void test_unit_aborted(utf::test_unit const& tu) override;         virtual void assertion_result(utf::assertion_result result) override;         // are the others required for my use case?     };

    test_observer                                    observer;

public:     test_observer_fixture();     ~test_observer_fixture();

public:     void setup();     void teardown(); };

and get called by my BOOST_DATA_TEST_CASE. What is the prefered way to use this Fixture to save the test result in case of failure/abort?

BOOST_DATA_TEST_CASE( wave,     utf_data::make_delayed<testsuite::dataset_loader>( "test_case/wave" ),     input, expected, orig_file_name) {    ...     auto [parse_ok, parse_result] = parse(input, parser, test_case_name);

    BOOST_REQUIRE(parse_ok);

    BOOST_TEST(parse_result == expected, btt::per_element());

       ??? my_diagnostic_saving .... (orig_file_name, input, parse_result);     ); }

Thanks, Olaf

Sorry for the late answer, In the teardown, you may check if the current test had failures. You can do so by accessing the global singleton "results_collector". If "tc" is the current test case (that you can get with "current_test_case"): test_results const& tr = results_collector.results( tc.p_id ); will give you the result. If the fixture should be applied to each of the test case inside your suite, you can install it like this: https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/tests_or... Accessing via the test result will let you check if this is a failure or an abort: the BOOST_TEST_REQUIRE (BOOST_REQUIRE is deprecated) will fire an abort if I remember well. Using this way to install the fixture will also let you access the instance of the fixture directly (see examples in the documentation). Since you solved your issue with the double messages, I would just use the appropriate fixture and forget about the observer. Raffi PS: I am not sure that the teardown is called for data test case (something I forgot to check). You may do that in the dtor of your fixture.

...

OK, my test_observer_fixture with overloads is ready:

class test_observer_fixture {      struct test_observer : public utf::test_observer      {          virtual void test_unit_aborted(utf::test_unit const& tu) override;          virtual void assertion_result(utf::assertion_result result) override;          // are the others required for my use case?      };

     test_observer                                    observer;

public:      test_observer_fixture();      ~test_observer_fixture();

public:      void setup();      void teardown(); };

and get called by my BOOST_DATA_TEST_CASE. What is the prefered way to use this Fixture to save the test result in case of failure/abort?

BOOST_DATA_TEST_CASE( wave,      utf_data::make_delayed<testsuite::dataset_loader>( "test_case/wave" ),      input, expected, orig_file_name) {     ...      auto [parse_ok, parse_result] = parse(input, parser, test_case_name);

     BOOST_REQUIRE(parse_ok);

     BOOST_TEST(parse_result == expected, btt::per_element());

        ??? my_diagnostic_saving .... (orig_file_name, input, parse_result);      ); }

Thanks, Olaf

Sorry for the late answer,

In the teardown, you may check if the current test had failures. You can do so by accessing the global singleton "results_collector". If "tc" is the current test case (that you can get with "current_test_case"):

test_results const& tr = results_collector.results( tc.p_id );

will give you the result.

If the fixture should be applied to each of the test case inside your suite, you can install it like this:

https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/tests_or...

Accessing via the test result will let you check if this is a failure or an abort: the BOOST_TEST_REQUIRE (BOOST_REQUIRE is deprecated) will fire an abort if I remember well. Using this way to install the fixture will also let you access the instance of the fixture directly (see examples in the documentation).

Since my global testobserver works for all events which may interested even in the future (e.g exceptions), I have problems to apply them in my use case. I've added a member function to my testobserer fixture void test_obserer_fixture::failure_diagnostic(std::string const&, ...)  which I can activate on failed tests using the events I've got from utf::observer::assertion_result() Now, in file test_main.cpp I have: ----------------------------------------- #define BOOST_TEST_MODULE "Test Suite" #include <boost/test/included/unit_test.hpp> #include <testsuite/test_observer_fixture.hpp> using test_observer_fixture = testsuite::test_observer_fixture; BOOST_TEST_GLOBAL_FIXTURE(test_observer_fixture); ----------------------------------------- and in (I have several)  concrete test case file: ------------------------------------------ #include <boost/test/unit_test.hpp> #include <boost/test/output_test_stream.hpp> #include <testsuite/test_observer_fixture.hpp> ... BOOST_AUTO_TEST_SUITE( my_testsuite ) BOOST_DATA_TEST_CASE( failure_test,...     ...,     input, expected, test_case_name) { ... } ------------------------------------------- and I can see/log: # assertion failed: failure_test/test_case/_0 But how to access the global fixture now? I've changed the test case to: ------------------------------------------ ... BOOST_DATA_TEST_CASE_F(test_observer_fixture, failure_test,     ...,     input, expected, test_case_name) {     ...     failure_diagnostic(...); } ------------------------------------------- Boost.Test ships a BOOST_DATA_TEST_CASE_F() macro for this case, which is a 'local' fixture, each time constructed and destroyed which is expensive - so  I like the idea of a global fixture to save to filesystem. Even I've installed it as global one too the events are doubled... but I can see that the test observer shows the test status followed by the call to failure_diagnostic - so the concept works. To reduce it: How can I access a member function of global fixture on a compilation unit layout shown above. Thanks, Olaf

On 18.08.18 20:51, Olaf Peter via Boost-users wrote:

Am 18.08.2018 um 20:38 schrieb Olaf Peter via Boost-users:

...

...

...

OK, my test_observer_fixture with overloads is ready:

class test_observer_fixture {      struct test_observer : public utf::test_observer      {          virtual void test_unit_aborted(utf::test_unit const& tu) override;          virtual void assertion_result(utf::assertion_result result) override;          // are the others required for my use case?      };

     test_observer observer;

public:      test_observer_fixture();      ~test_observer_fixture();

public:      void setup();      void teardown(); };

and get called by my BOOST_DATA_TEST_CASE. What is the prefered way to use this Fixture to save the test result in case of failure/abort?

BOOST_DATA_TEST_CASE( wave,      utf_data::make_delayed<testsuite::dataset_loader>( "test_case/wave" ),      input, expected, orig_file_name) {     ...      auto [parse_ok, parse_result] = parse(input, parser, test_case_name);

     BOOST_REQUIRE(parse_ok);

     BOOST_TEST(parse_result == expected, btt::per_element());

        ??? my_diagnostic_saving .... (orig_file_name, input, parse_result);      ); }

Thanks, Olaf

Sorry for the late answer,

In the teardown, you may check if the current test had failures. You can do so by accessing the global singleton "results_collector". If "tc" is the current test case (that you can get with "current_test_case"):

test_results const& tr = results_collector.results( tc.p_id );

will give you the result.

If the fixture should be applied to each of the test case inside your suite, you can install it like this:

https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/tests_or...

Accessing via the test result will let you check if this is a failure or an abort: the BOOST_TEST_REQUIRE (BOOST_REQUIRE is deprecated) will fire an abort if I remember well. Using this way to install the fixture will also let you access the instance of the fixture directly (see examples in the documentation).

Since my global testobserver works for all events which may interested even in the future (e.g exceptions), I have problems to apply them in my use case.

I've added a member function to my testobserer fixture

void test_obserer_fixture::failure_diagnostic(std::string const&, ...)  which I can activate on failed tests using the events I've got from utf::observer::assertion_result()

Now, in file test_main.cpp I have: ----------------------------------------- #define BOOST_TEST_MODULE "Test Suite" #include <boost/test/included/unit_test.hpp> #include <testsuite/test_observer_fixture.hpp>

using test_observer_fixture = testsuite::test_observer_fixture; BOOST_TEST_GLOBAL_FIXTURE(test_observer_fixture); -----------------------------------------

and in (I have several)  concrete test case file: ------------------------------------------ #include <boost/test/unit_test.hpp> #include <boost/test/output_test_stream.hpp> #include <testsuite/test_observer_fixture.hpp> ... BOOST_AUTO_TEST_SUITE( my_testsuite )

BOOST_DATA_TEST_CASE( failure_test,...     ...,     input, expected, test_case_name) { ... } -------------------------------------------

and I can see/log: # assertion failed: failure_test/test_case/_0

But how to access the global fixture now?

I've changed the test case to: ------------------------------------------ ... BOOST_DATA_TEST_CASE_F(test_observer_fixture, failure_test,     ...,     input, expected, test_case_name) {     ...     failure_diagnostic(...); } -------------------------------------------

Boost.Test ships a BOOST_DATA_TEST_CASE_F() macro for this case, which is a 'local' fixture, each time constructed and destroyed which is expensive - so  I like the idea of a global fixture to save to filesystem. Even I've installed it as global one too the events are doubled... but I can see that the test observer shows the test status followed by the call to failure_diagnostic - so the concept works.

To reduce it: How can I access a member function of global fixture on a compilation unit layout shown above.

even the use of

BOOST_FIXTURE_TEST_SUITE( my_testsuite, test_observer_fixture )

seems to create/destroy the fixture for each test case.

This one: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/tests_or... does it once per test suite. However it is isolated from the test case. The fact that a test case body can access the fixture is not a very good pattern and has several caveats (although very convenient). For instance, as you can observe, global fixtures and test case/suite fixtures do not have the same isolation. If you want to access the global fixture, you have to expose your fixture globally, provide some static member function that return a pointer to a singleton. The test framework itself will not create more than one instance. But I am a bit lost in what you are doing, and to me this does not go to the right direction. If the test module is very complicated to maintain, it reduces the chances to be useful, steady and stable on the long run. When I look back, I can see one of the comment in the code that says: // call diagnostic only in failure case and write then This means to me that a fixture should be executed for each test case. The purpose of the fixture would be to write the diagnostic in case of a failure. Does this only work for you? Best, Raffi