Re: RFC: GLib testing framework

[Stefan Kost <ensonic hora-obscura de>]

Hi Tim,

thanks for the work you put into this already!

First some comment regarding check. We use check in gstreamer and I use that in
buzztard too. I can understand that some people see an issue with having this as
a dependency. Besides also the api is a bit chaotic (no namespace prefix).
Besides that I do find the fork feature useful. It allows to run the full
testsuite, even if single test crash. This is usefule on buildbot-alike systems.
As check allows to swicth via envvar developers can easyly run tests without
forking.

Now some general comments. The autofoo test support should only be used as a
'launcher'. As Behadad pointed out, having single binaries for each test is
unmanagable. Check provides the following means to structure tests:
  TestBinary/TestSuite/TestCase/Test
I buzztard I mapp that to
  Component/Object/Aspect/Test
For e.g. gtk+ package
  "Component" would be gdk,gdk-pixbuf,gtk
  "Object/Module" would be GtkButton or GdkScreen
Keeping Aspects separated is imho really useful. The Aspects I have identified
and use so far are:
  "Positive Test/Example"
    * use the api in a normal way, if that fails, it certainly breaks stuff
    * also serves as example (that can be included in docs)
  "Negative Test"
    * try to break things (call things with out of bound parameters)
  "Performance"
    * these tests usually don't have clear asserts,
      but they should help to see if a change has impact on the performance
When designing new api, its been useful for me to write the example tests + a
skelleton api. Of course the tests initially fail, as the skelleton
implementation return NULL or FALSE. As stuff gets implemented, tests turn
green. This is called test driven development. I try to write the negative tests
in parallel, but it can be useful to let others write them (if the don't see the
implementation, but only the api docs, then this is black box testing).
Another benefit from this is to selectively run one aspect. This is useful for
monitored testing. Just running the test suite tells you that tests pass and
some fail. We can do more. We can generate coverage reports, we can run tests
under valdgrinds memcheck (or new threadcheck), we can run tests again with
different nicelevel. But we unlikely run all tests. Is not so useful to run the
performance tets under valgrind or under coverage.

As you propose to have hierarchical Suites all these usecase can be supported. I
see no point in making it mandatory (except that you can filter modes
explicitely on the commandline via -m) , but can warmly welcome to structure
tests in a hierarchy similar to the sources and use aspects.

Now some comments about the API
g_test_create_case -> g_test_case_create
g_test_create_suite -> g_test_suite_create

I would not export g_test_trap_fork(), it should be a parameter to the test
runner wheter it forks all tests or not. Does -k (keep-going) implies that tests
are forked?

In gstreamer we have some useful check extensions. Among those are mechanisms to
tests for asserts.

Would it make sense to allow overridable test_result loggers. This way a ide can
drive the test and gather result without the need to parse stdout.

One problem with unit-tests is that one can only tests the public api. The can
sometimes be worked around a bit with providing mock obejects. I am wondering if
 it would be useful to have internal tests inside the real code, call this from
the tests (maybe a separate aspect) and wrap those internal function with some
#ifdef so that is can be ommited by default from releases.

Finally, how would the logger output look like. check support a normal and
verbose mode. Providing similar formated logs would give us instant reports on
http://build.gnome.org

Okay, before my mail exceeds yours, I'll stop here,

Stefan