Re: New GnomeGoal proposal: InstalledTests

[Martin Pitt <martin pitt ubuntu com>]

Colin Walters [2013-04-25 10:45 -0400]:
> Now for things like NetworkManager that jhbuild is incapable of
> handling, I can certainly imagine how uninstalled tests would be very
> convenient for developers.

I fully agree, and I don't think there is much reason to drop
uninstalled tests in favor of installed ones; In most cases it should
be easy to support both. That's the model how development (should)
work, after all: You write a test for a new feature, then implement
it, and run "make check" (or "sudo make check") until it works; it
would be a step backwards to require installing the tests (and
clobbering your system) first.

> And it would likely make sense for some of those tests to support
> running them both installed and uninstalled.

I think this "installed" vs. "uninstalled" distinction should apply
to both the tests, as well as the test subject. automake has this
standard target "installcheck" which should mean "test the installed
binaries, not the ones from the build tree". All four combinations are
useful:

 (BB) build tree tests on build tree binaries: normal mode of development

 (BI) build tree tests on installed binaries: that's how e. g.
 Debian's autopkgtest works, and verifies that "make install" and the
 packaging actually works (e. g. you put stuff in the right place and
 didn't forget to install files); this is also useful to check how
 your potentially older installed version of the software holds up
 against your refined tests in trunk.

 (IB) installed tests on build tree binaries: verify backwards
 compatibility, i. e. whether you broke API/behaviour

 (II) installed tests on installed system: regular verification of
 daily image builds, self-check of installed systems and ostree
 snapshots

So it would be good if we can have all four and provide standard ways
of running them.

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)