Re: reftests

[Benjamin Otte <otte gnome org>]

On Fri, May 6, 2011 at 12:04 PM, Kristian Rietveld <kris gtk org> wrote:
> Which raises another question, would it be a good idea or make sense to merge the image differ into the GTK+ test utils so that other tests (e.g. the tree view scrolling test suite) can make use of it?
>
I did not spend any time thinking about generalizing code, but I guess
it would certainly make sense to merge the image diffing or even the
glade file => cairo surface containing snapshot code. So if you need
something like that, better generalize it then copying it or inventing
your own nefarious scheme.

> 1) There seems to be some logic behind where frequently run and less frequently tests are located.  Quoting from [1]:
>
> ``
> 1) Figure a place for the test case. For this it's useful to keep in mind
>   that make check will traverse CWD recursively. So tests that should be
>   run often when glib, gdk or gtk changed should go into glib/glib/tests/,
>   gtk+/gtk/tests/ or gtk+/gdk/tests/. Tests more thorough or planned to
>   be run less frequently can go into glib/tests/ or gtk+/tests/. This is
>   e.g. the case for the generic object property tester in
>   gtk+/tests/objecttests.c. To sum up:
>     glib/tests/                # less frequently run GLib tests
>     glib/glib/tests/           # frequent GLib testing
>     glib/gobject/tests/        # frequent GObject testing
>     gtk+/tests/                # less frequently run Gdk & Gtk+ tests
>     gtk+/gdk/tests/            # frequent Gdk testing
>     gtk+/gtk/tests/            # frequent Gtk+ testing
> ''
>
> I think this only makes sense if we want to continue to maintain this division between frequently and less frequently run tests.
>
What is a "frequently run" test? I can tell you that no treeview test
qualifies as "frequently run" to me when I'm hacking on GtkPaned. And
I think the same applies to you vice versa. Which reduces the
"frequently run" tests to a number very close to 0 I guess.

> 2) Tests that are maintained outside the normal source tree could be subject to becoming unmaintained quite quickly [2], which in turn leads to tests being run less often.  However, if the tests will still live under gtk+/unittest/ and are part of the default full build, then this might not be that big of a problem.
>
I've heard that argument before and the only response it got from me
was a confused look. Testsuite code is run regularly by make distcheck
and buildbots. If you add to this the fact that the code will keep
running unmodified until we break API again, this argument quickly
stops making any sense. At least to me. So building tests only as part
of make check (like everyone else does) seems like a very sound thing
to do, in particular for code as stable as GTK.

> The only issue I have with tests/ is that non-automated and automated tests might be intermixed, which is IMHO confusing.  Perhaps a subdirectory unittest/ under tests/ would work?
>
I would personally move testgtk and friends into demos/tests/ but I
don't care either way. I guess Matthias is the one that gets to decide
here.

Benjamin