Re: Some shortcomings in gtestutils

[Emmanuele Bassi <ebassi gmail com>]

hi Federico;

On 21 February 2013 02:46, Federico Mena Quintero <federico gnome org> wrote:

> So, I've been learning what gtestutils provides.  It is not bad, but it
> seems pretty pedestrian on some counts.  These are some things I'd like
> to change, or at least for someone to point me in the right way of doing
> them:
>
> * Tests abort as soon as a test fails, since tests ultimately depend on
> variations of g_assert().  This is fine in that it makes the "ensure
> things are okay" code look the same as "generic sanity check" code.
> However, it also means that a test suite aborts as soon as it encounters
> the first test that doesn't pass.  There is g_test_fail(), but the
> documentation pretty much only says, "You can use this function if your
> test failed in a recoverable way."  I don't know if that means that the
> test couldn't find a data file (but can continue anyway), or if it means
> "call this if you want the test to fail but want the test suite to
> continue running".

Matthias, Colin, Dan, and I were discussing this in the bug to add TAP
support for GTest, to get decent report harnesses instead of our
homegrown test report XML hack:

https://bugzilla.gnome.org/show_bug.cgi?id=692125

in short, the only way to safely run these tests in C is to execute
each unit inside a separate process; it's not a trivial change,
though.

for Clutter, we do compile all units inside the same binary (to cut
down compilation and link time), and then execute each unit
independently; we still abort() as soon as one unit asserts, but we
can easily change that and keep running. I also added a skip() and a
todo() wrappers, but that was just for my convenience.

> * It's hard to get a debugger up in the middle of "make check".  I can't
> think of a way to do it other than manually inserting code to sleep in
> the middle of the faulty test, and then attaching with a debugger
> manually.  Maybe having an environment variable or something, so that I
> could run
>
>   $ G_PAUSE_ON_FAILED_TESTS=1 make check

you can use TEST_ENVIRONMENT to set up the test environment variables
and command line arguments. it's standard automake. I use it to pass
the backend to the ABI check script, so that it's aware of which kind
of ABI should be available.

ciao,
 Emmanuele.

--
W: http://www.emmanuelebassi.name
B: http://blogs.gnome.org/ebassi/