Tests, demos, and all that

[Owen Taylor <otaylor redhat com>]

GTK+ now has a large number of non-installed executables scattered all
over the place:

Tests/demos:

 gdk-pixbuf/demos/pixbuf-demo.c

 gdk-pixbuf/testanimation.c
 gdk-pixbuf/testpixbuf-drawable.c
 gdk-pixbuf/testpixbuf-scale.c
 gdk-pixbuf/testpixbuf.c

 gtk/simple.c
 gtk/testgtk.c
 gtk/testcalendar.c
 gtk/testdnd.c
 gtk/testselection.c
 gtk/testinput.c
 gtk/testrgb.c
 gtk/testtext.c

test/demo/benchmark:

 gdk-pixbuf/pixops/timescale.c

Non-functioning:

 gtk/testthreads.c

Automated tests:

 gtk/testtextbuffer.c
 gdk-pixbuf/test-gdk-pixbuf.c

Examples from tutorial:

 examples/**.c

I'd like to bring some order to this chaos. With gdk-pixbuf
integration some reorganization is going to be needed in any case.
The gdk-pixbuf examples can't be built in the gdk-pixbuf/ directory
because they depend on GTK+ which won't be built until later.


he general structure I'd propose is:

 tests/ - automated tests
 demos/ - non-automated demos
 examples/ - examples from the tutorial

The first step would be to simply to move the above programs to fit
this directory structure. In the longer term, there are some other
issues that I think should be addressed:

 - testgtk badly needs to be split into multiple files and cleaned
   up. Right up testgtk mostly is a 'demo' but has various tests
   embedded in it that don't really fit this model and actually look
   like bugs.

   ("Foo" handlebox in the handlebox test, "Reparent Out" button in
   scrolled window test, etc.)

   I'd like to see testgtk cleaned up and made into a nice demo of
   GTK+. We should replace the buttons with a tree or list, (a
   GtkTLView test...), and add descriptions for each test.

   The Tk widget demo feature where each test has a button to show the
   source code is nice too, and shouldn't be hard to implement if we
   break up testgtk into one file per test.

 - The examples/ directory doesn't get built as part of the build
   process and has ad-hoc Makefile's instead of automake.

   I guess the intention of the makefiles in there is that they can be
   copied out somewhere else and used without modification.

   But I don't think that advantage outweighs the more substantial
   advantages of having the examples controlled by a Makefile.am:

   - We make sure that they continue to build
   - The programs can be tried out in place without installing
     GTK+.

   If we changed over the build process for the examples, we probably
   should flatten the directory structure. From past experience
   (with the GIMP expecially), having a large number of Makefile.am's
   can make the build process really crawl.

 - We have no structure in place for testing of GTK+ other than simply
   bringing testgtk up and trying all the tests.

   Fixing this is definitely a longer-term thing than the other
   problems.

   I'm not sure how much scripted-testing of the sort that Michael
   Meeks posted is going to help. There are just too many variables to
   get that going well, and judging the results is even more
   complicated.

   I suspect a better approach is going to be specially written
   torture tests that excercise widgets (or GTK+ subsystems)
   programatically. My general sense of past GTK+ defects is that they
   are:

    - generally triggered via application code rather than
      user actions.
    - mostly signaled by warnings or segfaults rather
      than misbehavor or misdisplay.

   Going through open and closed GTK+ bugs and classifying in terms
   of:

    - Are they platform specific or general?
    - Did they require special code to trigger them or could
      they be triggered through existing GTK+ tests?
    - What was the failure mode?

   Could be quite useful in getting a better QA scheme in place for
   GTK+.

   (Actually, I'm pretty happy in general about the quality level
   we've maintained for the stable releases of GTK+. There have not
   been many stupid bugs introduced in stable releases, but I still
   think we can do better.)

Regards,
                                               Owen