Re: RFC: GLib testing framework

[Stefan Kost <ensonic hora-obscura de>]

Hi Tim,

Tim Janik schrieb:
> On Thu, 1 Nov 2007, Stefan Kost wrote:
> 
>> Hi Tim,
> 
>> Now some comments about the API
>> g_test_create_case -> g_test_case_create
>> g_test_create_suite -> g_test_suite_create
> 
> i think this is a bit of a philosophy issue. i'd like to think about
> the new testing framework as one integrated thing, and that allows creation
> of suites and cases (that leads to the names i suggested).
> alternatively, GTestCase and GTestSuite can be viewed as self standing
> objects with API, which would lead to the names you suggest.
> 
> due to the very limited scope of the object APIs and the tight coupling
> with the rest of the suite for actually running suites/cases, i'm more
> inclined to the former naming/view though ;)

I though its more future proof and more aligned with what names a developer
would expect.

> 
>> I would not export g_test_trap_fork(), it should be a parameter to the
>> test
>> runner wheter it forks all tests or not. Does -k (keep-going) implies
>> that tests
>> are forked?
> 
> no, the forking API is used only by test developers and only for tests that
> are supposed to fail or run asyncronously. i have parts of that already
> implemented, so these are working examples that use the forking API:
> 
<snip>
> 
> the idea is to let the actual test binaries never do forks unless the above
> API is used (but then, it's the test writer who forks out) and to let the
> runner always fork+exec the test binaries. that way, test binaries stay
> very simple and straight forward according to the test implementation,
> so can be easily debugged in gdb.
> 
> then, if launched via the runner, if a test binary actually fails a test
> (and thus needs to abort because we can't catch exceptions in C), the
> runner
> can fork+exec a new instance of the test binary to execute the next test,
> in case --keep-going was specified.
>
So the runner forks the whole top-level-suite and watches the results listens to
SIGSEV? If that signal is triggered, it restarts the testsuite so that it
continues with what would be the next test according to th received results?

That would sound good.

> 
>> Would it make sense to allow overridable test_result loggers. This way
>> a ide can
>> drive the test and gather result without the need to parse stdout.
> 
> we intend to either allow multiple runner implementations (e.g. gtester
> for the
> command line, and another for ide integration), or to simply interface
> with the
> xml logging output that gtester is supposed to produce.
> 
>> One problem with unit-tests is that one can only tests the public api.
>> The can
>> sometimes be worked around a bit with providing mock obejects. I am
>> wondering if
>> it would be useful to have internal tests inside the real code, call
>> this from
>> the tests (maybe a separate aspect) and wrap those internal function
>> with some
>> #ifdef so that is can be ommited by default from releases.
> 
> i think at the very least, we'll need some hooks into Gtk+ for things like
> installing mock object vtable. my current idea is to wrap that up like:
> 
> #ifdef    GTK_INTERNAL_ABI
>   void gtk_test_install_widget_vtable (GtkTestVTableWithManyHooks *vtable);
> #endif

GObject could even have a test-iface that allows to have a generic gobject
testsuite to introspect and add the internal tests of a gobject.

> 
> so it's not part of the public API, but can be used by test binaries built
> inside the gtk+ package tree which link against libgtk-2.0.so.
> but this isn't part of the glib testing framework, and as i said, i'll
> send out an email on gtk testing routines at another point.

Beyond the api, it migh make sense to gather best practises and maybe even
provice to GObjectTest utillity stuff.

> 
>> Finally, how would the logger output look like. check support a normal
>> and
>> verbose mode. Providing similar formated logs would give us instant
>> reports on
>> http://build.gnome.org
> 
> i'd ideally like gtester to supply all valuable information in an xml file,
> so we can implement arbitrary report generators on top of that. there are
> so many different report targets and needs by various people, that i think
> we can only cover all if people can plug their report generators.
> 
>> Okay, before my mail exceeds yours, I'll stop here,
> 
> heh ;) thanks for the input.
> 
>> Stefan
> 
> ---
> ciaoTJ

Cu
  Stefan