Re: Gtk+ unit tests (brainstorming)

[Tim Janik <timj imendio com>]

On Thu, 26 Oct 2006, Michael Urman wrote:

> On 10/25/06, Havoc Pennington <hp redhat com> wrote:
> > Testing those is like testing segfault handling, i.e. just nuts. The
> > behavior is undefined once they print. (Well, for critical anyway.
> > g_warning seems to be less consistently used)
>
> Certainly setting out to test all critical cases would not add value
> corresponding to the effort; criticals are a different beast I
> shouldn't have included. Even for warnings, in certain cases making
> error cases testable would slow down real life performance without
> benefit. But preemptively deciding it's always impossible to test
> resilience of certain known warnings is a misstep. An option like
> -Werror is really useful, but hard wiring -Werror is too limiting.

the analogy doesn't quite hold. and btw, i'm not a fan of -Werror fwiw ;)

> Warnings especially, by not being criticals, imply a contract that the
> call will function reasonably (not necessarily "correctly") even
> during incorrect use. If this is not tested, it will not be correct.

this is not quite true for the GLib context. once a program triggers any
of g_assert*(), g_error(), g_warning() or g_critical() (this is nowadays
used for the implementaiton of return_if_fail), the program/library is
in an undefined state. that's because the g_log() error/warning/critical
cases are widely used to report *programming* errors, not user errors
(g_log wasn't initially designed with that scnerio in mind, but this is
how it's uses efefctively turned out).

so for the vast majority of cases, aborting on any such event and fail a
test is the right thing to do.

that doesn't imply no tests should be present at all to test correct
assertions per-se. such tests can be implemented by installing g_log
handlers, reconfiguring the fatality of certain log levels and by
employing fork-ed test mode (i adressed most of these in my original
email to some extend). these kind of tests will be rare though, and
also need to be carefully crafted because specific assertions and checks
can simply be optimized away under certain configurations, so relying
on them is of questionable merit. you might want to read up on
G_DISABLE_ASSERT or G_DISABLE_CAST_CHECKS on this for more details.

also, experience (s/experience/bugzilla/) tells us that the majority
of bugs reported against GLib/Gtk+ is not about missing assertions
or errors, which provides strong hinting what kind of tests you'd
want to put your focus on.

> --
> Michael Urman

---
ciaoTJ