Re: [gnome-print] Filtering code

[Scott Gifford <sgifford suspectclass com>]

Chema Celorio <chema@ximian.com> writes:

> Scott, I was thinking about the regression test suite for the
> filters.  Would it be better/possible if we integrated your tests
> into a sequence in generate.c rather than having a different
> infrastructure for testing the filters? the testing is very similar.

generate and filtertest actually work fairly differently---generate
pretty much takes a command line that says "run this test with this
backend", and filtertest takes a command line that says "encode stdin
with this sequence of encoders and this buffer size", so generate's
command-line parsing would have to be rewritten to accommodate
filtertest as it is written now.  I tried several testing strategies,
and found that using various encoding sequences, block sizes, and
files caught lots of bad interactions, so I'm pretty confident that
this is the right strategy for exhaustive, thorough testing of
filters.

The image tests in generate.c also exercise the filter code, and that
is probably enough for a reasonable test.  The filter tests execute
1480 individual tests (!!!) with different variations of parameters,
to test every nook and cranny of the filter code, and if we're just
using one filter sequence with one blocksize, that is complete
overkill.

It also probably makes sense to leave the thorough filter test suite
in its own directory.  This way, somebody who just wants a working
libgnomeprint doesn't have to sit through 10 minutes of filter tests,
but a developer making changes to the filter code can still run the
thorough filter tests to make sure they didn't break anything.  This
will be especially important if we add support for hints from the
printer driver/application/user which could cause different settings
and blocksizes to be used.

Some of the code in tests/filters/t/test might be useful.  It has a
more uniform interface to the testing scripts than run-test.pl---the
test must exit 0, and must have stdout output identical to what we
expect.  I use a technique similar to this for another program of mine
(StarTalk), where each test is a small shell script, and have found it
to be very easy and maintainable.  With each test in its own script
and a good test harness, it's easy to run just one or several tests,
and to run tests in groups (for example, "run all test related to
images", "run basic tests", "run very thorough tests", etc.).

I'm a huge fan of regression tests, and would be happy to do some work
in this area.  I think that the most straightforward tests for
gnome-print would be to have the correct postscript/PDF/metafile
output for each of several tests, run generate with appropriate
parameters, and make sure it succeeds (exits 0) and that the output is
identical to the known-good file.  If development changes the output,
the actual output and expected output can be compared visually (in
ghostscript, acroread, etc.), and if the new output file is correct,
the known-good output would be replaced with the new results.  That's
easy to automate, so that on a new platform, running "make check" can
verify that things are working like they're supposed to, and so that
before spinning a new release, a thorough test can make sure things
are basically working, without depending on the eyes of a tired
programmer to spot 1-pixel differences in viewed PS files.  Is there
any reason this won't work?

----ScottG.