[Test] redirecting unit test report

At 8:25 PM -0400 8/19/05, Gennadiy Rozental wrote:

...

In 1.33, the descriptive text is printed to std::cerr

I think you are mistaken. Log entries by default should appear in std::cout.

Oh drat! The test case in question is announcing failure via an assert(). That's where the unwanted output to std::cerr is coming from. Sorry for the bogus report.

...

Now, I never really liked redirecting stderr to a file in order to capture the report. What if something in the test program were to write to stderr? (Which is exactly what is happening now.) I see that there is now (in 1.33) a mechanism for setting the stream used by the report generator (boost:unit_test::results_reporter::set_stream()). But after spending the last couple of days pouring over the documentation and sources for Boost.Test, I have no idea where I would put a call to that function that doesn't involve patching the Boost.Test sources. There doesn't seem to be a place to insert such user code, at least not that I can find. Have I overlooked something, or is this really a hole in the present design / implementation?

There several thing you could do:

1. Check your redirection. Log should appear in std::cout. While report should appear in std::cerr 2. You could set a --log_level=nothing (witch is a good idea for automated test runners anyway) and no log entries will appear

My test runner allows the log level to be specified. When running it by hand (rather than from a chron job or the like), it can be helpful to see the log information go by on the screen, both from a progress reporting point of view, and also for immediate error context.

3. You could redirect log and/or report streams. Put configure testcase as a first one in a module:

BOOST_AUTO_TEST_CASE( configure ) { boost::unit_test::results_reporter::set_stream( report ); boost::unit_test::unit_test_log::set_stream( log ); }

Eew! What if somebody decides to use the new random order feature of the test framework? I guess this is a usable workaround for now, but I really think there needs to be some capability to insert user code into the process in a more controlled fashion.

At 3:03 AM -0400 8/20/05, Gennadiy Rozental wrote:

From results_reporter standpoint it doesn't matter. It only kicks in once all test cases are run.

Oh, right. So this will work for my specific problem, of changing the report stream, but wouldn't be a reliable way of setting the log stream. I think the point where user code should be insertable is in the main() function supplied by the unit test framework (defined in impl/unit_test_main.ipp). I would suggest splitting that function into two parts, by moving most of the contents of the try-block into a separate function, which I'll call unit_test_main for now. It would be: int unit_test_main(int argc, char* argv[]) { framework::run(); results_reporter::make_report(); return runtime_config::no_result_code() ? boost::exit_success : results_collector.results (framework::master_test_suite().p_id).result_code(); } and add an extern declaration for it to some appropriate public interface header (perhaps unit_test.hpp). The existing unit_test main() function is then changed to call a user substitutable function which is required to call unit_test_main(). I suggest that specific cut point in the unit_test main because it means that user code is run after the standard framework configuration has been performed, but before any actual work has been done. It also allows the user code to be wrapped around the actual work, making it easy to do cleanup on exit. A different way to do it that doesn't require a new function that the user code must call would be to insert a call to a user substitutable function between the framework::init() and framework::run() calls. Cleanup of user stuff could be done via a global scoped pointer that gets set to a cleanup object by the user code. I think that's a bit more complicated and messy though, so I'd prefer the wrapper function approach. Note that the configuration test case approach you suggested has the same messiness. The complicated part is presumably how to make that user substitutable function actually substitutable without making it overly complicated to do so or to not. At least, that's my understanding from this part of your message:

It may worth investigating, but keep in mind that it should be something that framework picks up automatically, but on the other doesn't require. These requirements usually contradict each other. Have any ideas?

Is my interpretation correct? I can think of a couple of ways to attempt to implement the user substitution. However, I don't claim to be an expert on what tricks will actually work across a broad set of tool chains. (In fact, I would claim to be an utter novice in this regard.) One relies on linker behavior, and from some web searching I suspect it isn't very portable (it works for gcc, but probably not for a lot of other linkers). In this approach, call the substitutable function unit_test_user_main. Put a declaration in some public header file. Add a .cpp file containing a trivial definition which just returns the result of calling unit_test_main, and include that in the unit test library. Have main call it in the appropriate place. Tell users to add behavior by providing their own definition, and ensuring that it appears earlier than the unit test library in the link list. The other way looks something like the following (ignoring syntax errors and other similar blunders). Is there anything wrong with this approach? Or is my naivete showing? It relies on the variable initializer being executed before the (optional) user's configuration constructor, but I would expect that initializer to actually be executed at link time. // in public header file extern int (*unit_test_user_main)(int argc, char* argv[]); // in framework implementation file int default_user_main(int argc, char* argv[]) { return unit_test_main(argc, argv); } int (*unit_test_user_main)(int argc, char* argv[]) = default_user_main; // in user file implementation file class my_config { my_config() { unit_test_user_main = my_main; }; static int my_main(int argc, char* argv[]); }; int my_config::my_main(int argc, char* argv[]) { // do user-specific setup return unit_test_main(argc, argv); } my_config c;

At 1:25 AM -0400 8/23/05, Gennadiy Rozental wrote:

[...]

...

The existing unit_test main() function is then changed to call a user substitutable function which is required to call unit_test_main().

That is a major drawback - we couldn't rely on user concience.

As I described things, the default (i.e. if the user doesn't provide a substitute) would just call unit_test_main, so that the right thing happens if the user does nothing in this area. It doesn't seem unreasonable to me to require that if a user *does* provide a substitute, that this substitute must call unit_test_main, i.e. make that part of the substitution contract. On the other hand, your global fixture approach looks fine too, and is probably easier to document. So I'd say go with that.

"Kim Barrett" <kab@irobot.com> wrote in message news:p06230900bf3241d57905@[192.168.161.194]...

At 1:25 AM -0400 8/23/05, Gennadiy Rozental wrote:

...

template<typename F> class global_fixure_impl : public global_fixure { virtual void setup() { new (m_space) F; } virtual void teardown() { ((F*)m_space)->~F(); }

char m_space[sizeof(F)]; };

Alignment of m_space must be appropriate for the alignment of F. I think this can be addressed by putting m_space in a union with a value whose type is the result of boost::type_with_alignment<boost::alignment_of<F>::value>::type.

char m_space[sizeof(F)]; is going to be aligned properly automatically Gennadiy

"Kim Barrett" <kab@irobot.com> wrote in message news:p06230904bf327ef3cc00@[192.168.161.194]...

At 12:38 PM -0400 8/24/05, Gennadiy Rozental wrote:

...

"Kim Barrett" <kab@irobot.com> wrote in message news:p06230900bf3241d57905@[192.168.161.194]...

...

At 1:25 AM -0400 8/23/05, Gennadiy Rozental wrote:

...

template<typename F> class global_fixure_impl : public global_fixure { virtual void setup() { new (m_space) F; } virtual void teardown() { ((F*)m_space)->~F(); }

char m_space[sizeof(F)]; };

Alignment of m_space must be appropriate for the alignment of F. I think this can be addressed by putting m_space in a union with a value whose type is the result of boost::type_with_alignment<boost::alignment_of<F>::value>::type.

char m_space[sizeof(F)]; is going to be aligned properly automatically

I don't see how. See analogous discussion recently on boost developers list under subject "alignment problem in proposed any alternative", in particular the message from Martin Bonner on 8/11/05: http://aspn.activestate.com/ASPN/Mail/Message/boost/2772775 Otherwise, what would be the point of boost::aligned_storage (which actually packages up the union idiom I was suggesting be used)?

You maybe right. I will use new then, virtual void setup() { m_space = new char[sizeof(F)]; new (m_space) F; } virtual void teardown() { ((F*)m_space)->~F(); delete m_space; } char* m_space; Gennadiy

...

virtual void setup() { m_space = new char[sizeof(F)]; new (m_space) F; } virtual void teardown() { ((F*)m_space)->~F(); delete m_space; } char* m_space;

In that case, why use m_space at all. Instead, just something like the following (which may have stupid syntax errors):

template<typename F> class global_fixture_impl : public global_fixture { virtual void setup() { m_f = new F; }; virtual void teardown() { delete m_f; m_f = 0; }; F* m_f; }

Yes, you right of course ;)) Gennadiy