Re: [gnome-print] Filtering code

[Chema Celorio <chema ximian com>]

On Tue, 2002-06-18 at 23:04, Scott Gifford wrote:
> Chema Celorio <chema@ximian.com> writes:
> 
> > Scott, I was thinking about the regression test suite for the
> > filters.  Would it be better/possible if we integrated your tests
> > into a sequence in generate.c rather than having a different
> > infrastructure for testing the filters? the testing is very similar.
> 
> generate and filtertest actually work fairly differently---generate
> pretty much takes a command line that says "run this test with this
> backend", and filtertest takes a command line that says "encode stdin
> with this sequence of encoders and this buffer size", so generate's
> command-line parsing would have to be rewritten to accommodate
> filtertest as it is written now.  I tried several testing strategies,
> and found that using various encoding sequences, block sizes, and
> files caught lots of bad interactions, so I'm pretty confident that
> this is the right strategy for exhaustive, thorough testing of
> filters.

Ok

> The image tests in generate.c also exercise the filter code, and that
> is probably enough for a reasonable test.  The filter tests execute
> 1480 individual tests (!!!) with different variations of parameters,

wow.

> to test every nook and cranny of the filter code, and if we're just

excellent.

> using one filter sequence with one blocksize, that is complete
> overkill.

Yeah


> It also probably makes sense to leave the thorough filter test suite
> in its own directory.  This way, somebody who just wants a working
> libgnomeprint doesn't have to sit through 10 minutes of filter tests,
> but a developer making changes to the filter code can still run the
> thorough filter tests to make sure they didn't break anything.  This
> will be especially important if we add support for hints from the
> printer driver/application/user which could cause different settings
> and blocksizes to be used.

I was thinking of adding flags to ./run-test.pl. We need a flag to test
embedding with all of the fonts in the system. So we could do
--full-fonts ,  --full-filters and --full-all. 

> Some of the code in tests/filters/t/test might be useful.  It has a
> more uniform interface to the testing scripts than run-test.pl---the
> test must exit 0, and must have stdout output identical to what we
> expect.  I use a technique similar to this for another program of mine
> (StarTalk), where each test is a small shell script, and have found it
> to be very easy and maintainable.  With each test in its own script
> and a good test harness, it's easy to run just one or several tests,
> and to run tests in groups (for example, "run all test related to
> images", "run basic tests", "run very thorough tests", etc.).

yep

 
> I'm a huge fan of regression tests, and would be happy to do some work

Me too ;-)

> in this area.  I think that the most straightforward tests for
> gnome-print would be to have the correct postscript/PDF/metafile
> output for each of several tests, run generate with appropriate
> parameters, and make sure it succeeds (exits 0) and that the output is
> identical to the known-good file.  If development changes the output,

I am testing the files generated with gs (ps & pdf) and acroread (pdf).

> the actual output and expected output can be compared visually (in
> ghostscript, acroread, etc.), and if the new output file is correct,
> the known-good output would be replaced with the new results.  That's
> easy to automate, so that on a new platform, running "make check" can
> verify that things are working like they're supposed to, and so that
> before spinning a new release, a thorough test can make sure things
> are basically working, without depending on the eyes of a tired
> programmer to spot 1-pixel differences in viewed PS files.  Is there
> any reason this won't work?

I think it is hard to do fine testsing, most of our bugs have been big
bugs rather than 1 pixel differences. I'm shooting to fix/identify the
big fishes first, then we can think about small things. But with all the
new stuff that gnome-print needs we need more works-fails tests than
quality or visual stuff.

regards,
Chema