Re: Some shortcomings in gtestutils

[Peter Hutterer <peter hutterer who-t net>]

On Wed, Feb 20, 2013 at 08:46:34PM -0600, Federico Mena Quintero wrote:
> Hi, everyone,
>
> I've been writing some tests for GtkFileChooserButton and putting them
> in gtk+/gtk/tests/filechooser.c - this is the old test suite,
> resurrected and alive.
>
> So, I've been learning what gtestutils provides.  It is not bad, but it
> seems pretty pedestrian on some counts.  These are some things I'd like
> to change, or at least for someone to point me in the right way of doing
> them:
>
> * Tests abort as soon as a test fails, since tests ultimately depend on
> variations of g_assert().  This is fine in that it makes the "ensure
> things are okay" code look the same as "generic sanity check" code.
> However, it also means that a test suite aborts as soon as it encounters
> the first test that doesn't pass.  There is g_test_fail(), but the
> documentation pretty much only says, "You can use this function if your
> test failed in a recoverable way."  I don't know if that means that the
> test couldn't find a data file (but can continue anyway), or if it means
> "call this if you want the test to fail but want the test suite to
> continue running".
>
> * It's hard to get a debugger up in the middle of "make check".  I can't
> think of a way to do it other than manually inserting code to sleep in
> the middle of the faulty test, and then attaching with a debugger
> manually.  Maybe having an environment variable or something, so that I
> could run
>
>   $ G_PAUSE_ON_FAILED_TESTS=1 make check
>
> when a test fails, I would get told, "attach to $pid now" or somthing.
> Maybe even have a G_NAME_OF_TEST_TO_PAUSE_IN variable to pause for a
> specific test, not any one that fails.

not sure how well this applies in your case, but what I do in the xorg
integration tests is to send a SIGSTOP to the server based on some
environment variables. then you can attach gdb, SIGCONT it and debug it.
(I don't do this on test failure but on server startup, but the principle
should work either way)

Cheers,
   Peter

> * The documentation on gtestutils is far from stellar :)
>
> * Now that the a11y infrastructure is included in-the-box with GTK+, it
> sounds like a good idea to start putting an a11y-based testing
> infrastructure in GTK+ itself.  For the file chooser's tests, I had to
> do some manual hacks to simulate, "click this button", "find the
> dialog", "click the dialog's button"... this is tricky C code that
> assumes too much knowledge of the internals of the toolkit, and I'd love
> to write some higher-level stuff instead for such tests.  (Feel free to
> reply about this in a different thread - this could derail *this* thread
> pretty fast) :)
>
> Thoughts?  We have a useful battery of tests now, and it only seems that
> improving the testing infrastructure could lead to people actually
> wanting to write more test code.
>
>   Federico
>
>
> _______________________________________________
> gtk-devel-list mailing list
> gtk-devel-list gnome org
> https://mail.gnome.org/mailman/listinfo/gtk-devel-list