Re: RFC: GLib testing framework

[Tim Janik <timj imendio com>]

On Thu, 1 Nov 2007, Stefan Kost wrote:

> Hi Tim,

> Now some comments about the API
> g_test_create_case -> g_test_case_create
> g_test_create_suite -> g_test_suite_create

i think this is a bit of a philosophy issue. i'd like to think about
the new testing framework as one integrated thing, and that allows creation
of suites and cases (that leads to the names i suggested).
alternatively, GTestCase and GTestSuite can be viewed as self standing
objects with API, which would lead to the names you suggest.

due to the very limited scope of the object APIs and the tight coupling
with the rest of the suite for actually running suites/cases, i'm more
inclined to the former naming/view though ;)

> I would not export g_test_trap_fork(), it should be a parameter to the test
> runner wheter it forks all tests or not. Does -k (keep-going) implies that tests
> are forked?

no, the forking API is used only by test developers and only for tests that
are supposed to fail or run asyncronously. i have parts of that already
implemented, so these are working examples that use the forking API:

/* prototype, updated from the initial proposal */
  gboolean g_test_trap_fork             (guint64              usec_timeout,
                                         GTestTrapFlags       test_trap_flags);

/* fork around a failing test and assert the intended cause of failure */
  if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDERR))
    {
      g_assert_not_reached();
    }
  g_test_trap_assert_failed();
  g_test_trap_assert_stderr ("*ERROR*should not be reached*");

/* fork out to interrupt the test after a specific timeout */
  if (g_test_trap_fork (1.5 * 1000000, 0)) // give it 1.5 secs at maximum
    {
      /* loop and sleep forever */
      while (TRUE)
        g_usleep (1000 * 1000);
    }
  g_test_trap_assert_failed();              // testing timeout logic here
  g_assert (g_test_trap_reached_timeout()); // ensure it failed due to timeout

/* fork out to assert stdout pattern */
  if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDOUT))
    {
      g_print ("some stdout text: somagic17\n");
      exit (0);
    }
  g_test_trap_assert_passed();
  g_test_trap_assert_stdout ("*somagic17*");


the idea is to let the actual test binaries never do forks unless the above
API is used (but then, it's the test writer who forks out) and to let the
runner always fork+exec the test binaries. that way, test binaries stay
very simple and straight forward according to the test implementation,
so can be easily debugged in gdb.

then, if launched via the runner, if a test binary actually fails a test
(and thus needs to abort because we can't catch exceptions in C), the runner
can fork+exec a new instance of the test binary to execute the next test,
in case --keep-going was specified.

> Would it make sense to allow overridable test_result loggers. This way a ide can
> drive the test and gather result without the need to parse stdout.

we intend to either allow multiple runner implementations (e.g. gtester for the
command line, and another for ide integration), or to simply interface with the
xml logging output that gtester is supposed to produce.

> One problem with unit-tests is that one can only tests the public api. The can
> sometimes be worked around a bit with providing mock obejects. I am wondering if
> it would be useful to have internal tests inside the real code, call this from
> the tests (maybe a separate aspect) and wrap those internal function with some
> #ifdef so that is can be ommited by default from releases.

i think at the very least, we'll need some hooks into Gtk+ for things like
installing mock object vtable. my current idea is to wrap that up like:

#ifdef	GTK_INTERNAL_ABI
  void gtk_test_install_widget_vtable (GtkTestVTableWithManyHooks *vtable);
#endif

so it's not part of the public API, but can be used by test binaries built
inside the gtk+ package tree which link against libgtk-2.0.so.
but this isn't part of the glib testing framework, and as i said, i'll
send out an email on gtk testing routines at another point.

> Finally, how would the logger output look like. check support a normal and
> verbose mode. Providing similar formated logs would give us instant reports on
> http://build.gnome.org

i'd ideally like gtester to supply all valuable information in an xml file,
so we can implement arbitrary report generators on top of that. there are
so many different report targets and needs by various people, that i think
we can only cover all if people can plug their report generators.

> Okay, before my mail exceeds yours, I'll stop here,

heh ;) thanks for the input.

> Stefan

---
ciaoTJ