Some shortcomings in gtestutils

[Federico Mena Quintero <federico gnome org>]

Hi, everyone,

I've been writing some tests for GtkFileChooserButton and putting them
in gtk+/gtk/tests/filechooser.c - this is the old test suite,
resurrected and alive.

So, I've been learning what gtestutils provides.  It is not bad, but it
seems pretty pedestrian on some counts.  These are some things I'd like
to change, or at least for someone to point me in the right way of doing
them:

* Tests abort as soon as a test fails, since tests ultimately depend on
variations of g_assert().  This is fine in that it makes the "ensure
things are okay" code look the same as "generic sanity check" code.
However, it also means that a test suite aborts as soon as it encounters
the first test that doesn't pass.  There is g_test_fail(), but the
documentation pretty much only says, "You can use this function if your
test failed in a recoverable way."  I don't know if that means that the
test couldn't find a data file (but can continue anyway), or if it means
"call this if you want the test to fail but want the test suite to
continue running".

* It's hard to get a debugger up in the middle of "make check".  I can't
think of a way to do it other than manually inserting code to sleep in
the middle of the faulty test, and then attaching with a debugger
manually.  Maybe having an environment variable or something, so that I
could run

  $ G_PAUSE_ON_FAILED_TESTS=1 make check

when a test fails, I would get told, "attach to $pid now" or somthing.
Maybe even have a G_NAME_OF_TEST_TO_PAUSE_IN variable to pause for a
specific test, not any one that fails.

* The documentation on gtestutils is far from stellar :)

* Now that the a11y infrastructure is included in-the-box with GTK+, it
sounds like a good idea to start putting an a11y-based testing
infrastructure in GTK+ itself.  For the file chooser's tests, I had to
do some manual hacks to simulate, "click this button", "find the
dialog", "click the dialog's button"... this is tricky C code that
assumes too much knowledge of the internals of the toolkit, and I'd love
to write some higher-level stuff instead for such tests.  (Feel free to
reply about this in a different thread - this could derail *this* thread
pretty fast) :)

Thoughts?  We have a useful battery of tests now, and it only seems that
improving the testing infrastructure could lead to people actually
wanting to write more test code.

  Federico