Re: build.gnome.org

[Colin Walters <walters verbum org>]

On Wed, 2013-01-16 at 07:39 +0100, Martin Pitt wrote:

> Right now in our Juju charm it's a manual list:
> 
>   http://bazaar.launchpad.net/~jibel/charms/quantal/jhbuild/trunk/view/head:/files/jhbuild.config/gnome-core.sysdeps
> 
> I'm not quite sure why; Jean-Baptiste, did jhbuild sysdeps not work
> well enough in principle?

I'd be the first to admit the jhbuild sysdeps stuff is still quite
hacky, but it's also a lot more sustainable and reproducible.

> Yeah, there are several modules which show a problem with the test
> environment; that, and e. g. g-settings-daemon is stumbling over
> "PYTHONPATH for python3"
> (https://bugzilla.gnome.org/show_bug.cgi?id=688353) etc.

Yeah, ugly =(  

> I fully agree; we need both unit and these system integration tests.
> I'd like to think that adding "make check" tests isn't conflicting to
> that; when writing tests with above in mind, you can usually set them
> up in a way that they can run both against the built source tree as
> well as the installed system; for example, when submitting the tests
> for gvfs I added both a "make check" as well as a "make installcheck"
> rule, and you can just run e. g.  tests/gvfs-test.py out of a pristine
> git checkout. In many cases that's usually a matter of not writing the
> tests with hardcoded paths against the source and then setting
> variables like $PATH, $GST_REGISTRY_1_0, etc. to the build tree when
> running the tests against the build tree.

Right, but this gets very ugly very fast...there are a lot of those
environment variables, they don't all have quite the same semantics,
etc.  jhbuild is pretty much the canonical reference set.

In this discussion let's use glib as our example, because:

1) glib is fundamentally necessary
2) It has a lot of tests
3) Where glib goes, so goes the rest of our ecosystem

> We also use those tests in our "autopkgtest" model where we install a
> proposed package update into a standard distro install and then run
> the package's tests against that system. Architecture wise this has
> the same requirements as for above "system-wide smoke tests".

Ah, so this is:
http://developer.ubuntu.com/packaging/html/auto-pkg-test.html

Pretty interesting.  But here's the thing - the glib example from that
page is pretty lame, because all of the "makecheck" tests that are
presently trapped inside the glib tree could do this much better, if we
liberated them to be installed tests.

What I hope is that for projects like glib, we simply change the vast
majority of tests that exist now to be installed.  What then replaces
the "make check" developer workflow is:

$ jhbuild make
$ (set -e; for x in ~/build/gnome/bin/tests/glib/*; do $x; done)

Or perhaps we could define a standard "run all the tests" binary name,
so that'd be:

$ jhbuild make
$ ~/build/gnome/bin/tests/glib/all-tests

Now what gets interesting about the installed model is that after
updating GLib, it's trivial to run the *pygobject* tests, without
arbitrarily rebuilding pygobject.  Particularly important if you care
about binary compatibility, as we do for glib, since you're testing new
GLib against old pygobject binaries:

$ jhbuild buildone glib
$ ~/build/gnome/bin/tests/pygobject/all-tests

If we'd had this infrastructure in place, Ryan would have immediately
seen gjs and pygobject's tests blow up after committing:
http://git.gnome.org/browse/glib/commit/?id=c2055f22f4399a23d1c02a94f8b029212e37e162

I was going to raise this proposal once I had the infrastructure in
place to implement it, which is hopefully by the end of this month, but
since you brought it up - let's figure out a plan.

There are some people just hacking up the Makefiles in OpenEmbedded:
http://patches.openembedded.org/patch/32063/
Clearly not long term sustainable.