Re: build.gnome.org

[Colin Walters <walters verbum org>]

On Tue, 2013-01-15 at 11:07 +0100, Martin Pitt wrote:

> We have experimented with that a bit, by building 
> 
>   https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/

Interesting!  Looks quite useful.  Are you doing anything with
respect to the "jhbuild sysdeps --install" infrastructure or is
the system package set maintained manually?

> So we currently have 159 modules of which some 30 are
> unstable/failing. This is not totally hopeless, but it would be good
> to at least get the libraries working (glib, gvfs, etc.)

Yeah, up to gtk+ at least.  

The gnome-control-center failure appears to be something in the
jenkins/jhbuild glue not understanding how to properly do git
submodules.

> Your http://ostree.gnome.org builds indeed look very stable, mostly
> because they aren't runningi checks.

That's one factor.  It's also not building apps.  A further nontechnical
factor is that I watch the ostree.gnome.org builds like a mother with a
newborn baby...

>  Do you eventually plan on doing
> this, or is OSTree not the right place for this?

So...tests.  An interesting and complex topic.  I've had a document
started about this for a while:

https://docs.google.com/document/d/1DXgGEKTbC4ed1DFW3Mu-48TpHtq-xdfhDtwoUOWWcHg/edit

> Our goal is to provide CI with not only build tests, but also runtime
> tests, so that we can immediately see if e. g. a glib commit breaks
> something in settings-daemon or pygobject. Do you think it makes sense
> to do that in ostree.g.o, or should we have separate
> "functionality/reverse dependency test" builds for those?

Pretty soon, once I finish up some infrastructure, I'll have the key
"smoke test" - does the system boot in a standard qemu configuration,
with no core dumps or fatal service errors, successfully logging into
gdm?

See:
http://git.gnome.org/browse/gnome-ostree/tree/src/ostbuild/js/builtins/qa_smoketest.js

This should all be pretty fast - I want this test running within 5
minutes after I push a commit to glib/gjs/whatever.  And 5 minutes is
slow - with some optimizations, that could be 1 minute.

*After* that test has completed, we can do stuff like run selected tests
from glib/pygobject/gjs etc.  In the installed test model, we can choose
exactly which one of those tests to run.

I know this installed tests model is orthogonal (somewhat complementary,
somewhat conflicting) to the work you've been doing for the current
"distcheck" model of nondestructive per-user tests.  It's true that
those types of tests can be very easy and fast for developers to run.  

But the qemu-based testing model is very powerful, and there's no excuse
for not having a suite of qemu-based tests run automatically build
server side.