Hello Raffi,
BOOST_AUTO_TEST_SUITE(....
BOOST_DATA_TEST_CASE( declaration, dataset.input() ^ dataset.expect() ^ dataset.test_case_name(), code, expect_AST, test_case_name) { using attribute_type = ast::declaration; auto const parser = parser::declaration;
x3_test::testing_parser
parse; auto [parse_ok, parsed_AST] = parse(code, parser); BOOST_TEST(parse_ok); BOOST_REQUIRE_MESSAGE(parse_ok, x3_test::report_diagnostic(test_case_name, code, parsed_AST) );
BOOST_TEST(parsed_AST == expect_AST, btt::per_element()); BOOST_REQUIRE_MESSAGE(x3_test::current_test_passing(), x3_test::report_diagnostic(test_case_name, code, parsed_AST) ); }
BOOST_AUTO_TEST_SUITE_END()
After some testing, the error count is twice the real (obviously here) but it's all above a work arround.
It seems, that BOOST_REQUIRE_MESSAGE evaluates always the message predicate (shown by using --log_level=all) nevertheless the test case successed or failed. If failed it is streamed to the console.
That sounds odd to me. Would it be possible to check if the messages get evaluated in case you do not pass --log_level=all ?
I reduced it and found the original problem. Checking parse_ok twice results into two messages, this explain what I faced (and misinterpreted in the last mail). The reason for introducing this was the intention of: - check on parse_ok success - if not, generate a diagnostic message - all further checks can be canceled - compare the result per_element with reference/gold data - if failed, generate a diagnostic message - note, report_diagnostic saves the parsed_AST for later inspection The problem may rise due to my limited understanding about the macros arround.
What about :
1- using test-case fixtures for writing things at the end of the test 2- registering an observer for getting the failure signal?
The observer is (kind of) described here: https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/fra...
You may install and remove it globally by using a global fixture:
https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_refe...
Once registered, it will capture the events that you want, especially the failure cases.
From the observer, you can get the current test cases with this
https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/fra...
I will check/evaluate it next time, it simplifies diagnostic since the parsed_AST can be written by the observer Thank you, Olaf