Re: jhbuild continuous integration testing: status and plans

[Colin Walters <walters verbum org>]

On Mon, 2013-03-11 at 11:37 +0000, Simon McVittie wrote:

> DEP8 tests can work like this with sufficient support from the package's
> upstream developer (telepathy-gabble-tests 0.17.x from experimental
> works like this). 

I looked this up from here:
http://packages.debian.org/experimental/telepathy-gabble

So basically there's a manually-created subpackage named "-tests", and
a .install file to install them, and a --enable-installed-tests
configure option.  But DEP8 doesn't cover this, so there's no
established conventions, right?

> Maemo's continuous integration system (CITA) enforced the approach you
> advocate, and my experience from that system is that it requires
> relatively intrusive changes to most packages to turn tests designed for
> "make check" into something you can run like that. 

I hadn't heard of CITA before, so I did a bit of google searching, and
the #3 result was this:

http://maemo.gitorious.org/maemo-af/libcontentaction/blobs/a28157e74d0331f3b7388f5ca98f20ee2d84148f/t/lca-cita-test

Which pretty clearly demonstrates what you're saying =)

> That's fine if it's
> something the package's upstream maintainer wants, but if they don't,
> maintaining it as a distribution patch is pretty awkward.

Right.  That's obviously the tricky part - convincing a large enough
subset of the audience here that they should support this.  I don't want
to carry patches (because that breaks continuous integration).

While 'make check' is OK for "white box unit tests" (i.e. tests that
call internal API), I'd say upwards of 80% of tests in GLib for example
are black box, and could just as easily be run installed.

If we can standardize conventions (--enable-installed-tests, install
to /usr/share/$pkgname/tests), then jhbuild could be updated to be aware
of them, and the new equivalent of "make && make check" could be:

$ jhbuild make && jhbuild check glib 

Note that "jhbuild make" actually runs "make install".  This is a bit
alien to "package people" because it's ***really*** important to note
that jhbuild operates on a sandbox, not your actual system.  It'd
obviously be very painful if you had to replace your system copy of glib
to run the tests, and I'm sure the lack of this sandbox in the dpkg
universe is why DEP8 is written the way it is.

Anyways, now with installed tests we have advantages over "make check";
for example, the flexibility to run say the gjs tests after updating
glib:

$ jhbuild make && jhbuild check gjs

Or in fact, just run all the tests against the current tree, without
changing anything:

$ jhbuild check

Now, what would be quite tedious is supporting *both* uninstalled and
installed, at least in the general case.  So I do realize that I have
an uphill battle here in some cases.

But my plan still remains to just change say gjs to use installed tests
at the start of the 3.10 cycle, implement a really basic "jhbuild
check" (and also a more elaborate version in the gnome-ostree build
system using qemu), and see how well it works.

That said - I'm just going to add the authors of DEP8 to this discussion
in the hopes of making it look more like "fully installed tests" (my
proposal does kind of lack a name, and that's the best I can come up
with at the moment...)