Re: [gnome-love] Tinderbox (automatic daily build) for GNOME

[Sergio Villar Senin <svillar igalia com>]

Hi,

Juan José Sánchez Penas wrote:
> Hi all :)
>
> On Tue, Mar 28, 2006 at 05:45:59PM +0200, Julien Gilli wrote:
>
> > On 3/28/06, Alejandro Garcia Castro <acastro igalia com> wrote:
> >
> > > I agree, we can investigate the most simple way to build gnome with
> > > mozilla
> > > tinderbox, we can also create some kind of roadmap from that first
> > > deployment in order to analyse what could we get from it. We could even
> > > deploy some kind of sandbox with a simple prototype example, just a couple
> > > of modules. After buildbot and mozilla-tinderbox tests we can decide. Does
> > > this suit with what you were planning Jaap, Jérôme, Julien?
> >
> > That sounds good to me.
>
> I work with Alejandro at Igalia, and am also interested on the topic :) We
> have already some very preliminary results we are going to share very soon,
> but I wanted to propose something first.

here at Igalia we have just released a first version of an automatic
build with tinderbox for GNOME. This is a very first version, with a lot
of things still to do, but we think it can serve as a proof.

You can take a look here: http://tinderbox.fisterra.org. As you can see
 we only build the required packages for glib listed in the gnome-2.14
moduleset.

This is how we did it:

1- Jhbuild changes
Our idea was to call jhbuild buildone for each module of the moduleset
into the tinderbox loop. In order to integrate the tinderbox loop with
jhbuild we had to be able to execute the different phases of a 'jhbuild
buildone' one by one.

So we modified some jhbuild files in order to add the following options
to the 'jhbuild buildone':
-u : just updates
-n : just configures
-b : just builds
-l : just installs

We're not python experts, so for the moment the patch is very nasty, it
just simply works, but we think that it lacks some design.

2- Tinderbox changes

There are not so many changes in the tinderbox client side. We just call
'jhbuild buildone' with the correspondent option for each build phase.
We have 5 tinderbox phases: print_environment (does not call jhbuild),
checkout, configure, build and install.

First of all we changed the build_shellscript because it iterates over
the trees passed as arguments. We do not want such behaviour, we want it
to iterate over the modules issued by the 'jhbuild list' command and
exactly in the same order. That's why we put in the build_shellscript a
code that iterates over the result of the 'jhbuild list' and calls the
whole tinderbox loop for each module passing the module name in the
variable that the tinderbox uses for passing the tree name.

The most important change is that we do not take into account the values
stored in the VC_TREE variable that comes from the buildcf file. We just
let jhbuild to do everything, we just need to call
jhbuild buildone -u $module
jhbuild buildone -c $module
jhbuild buildone -b $module
jhbuild buildone -l $module
inside the tinderbox loop (a call per phase) to have all the work done.

Talking about the server side of the tinderbox, there was a problem. The
server receives the logs of the builds from the clients by mail. Then it
checks that the specified log belongs to a list of valid trees that must
be stored in a server configuration file called TreeData.pm. As a
temporal fix, we wrote the trees by hand in that file, but it should not
be done that way. A possible fix will be to change the get_all_trees
function of the same file in order to allow it to return the trees
stored in the file plus a list of trees created dynamically from the
output of 'jhbuild list'.

Work to do from here:
   * Implement a good solution for enabling phase-by-phase execution of
jhbuild build one
   * modify the build_shellscript in order to allow the build of a
single module of the whole gnome tree (with a new parameter?)
   * Automatic build of VC_TREE variable (and also VC_TREE_GROUPS) based
on the output of jhbuild (jhbuild list and jhbuild info)
   * Authenticate the client logs received by mail, it could used any
kind of email signing
   * Bonsai integration: currently bonsai does not support remote
repositories. There are at least alternatives
      * Replace the bonsai by another tool that does support such
configuration like ViewCVS
      * Replace the notification script (dolog.pl)

These are the main advantages in our humble opinion, a system like this
will have over another alternatives:
   * integration with mozilla webtools: Bonsai, LXR ...
   * the clients and the servers could be located in different machines,
so the tinderbox server could be hosted into a GNOME machine, and the
tinderbox clients could be spread all over the world. This implies that
a policy should be defined in order to select the clients for the builds.
   * it could allow the building of a remote distributed compilation
system. That's why it uses the email as transport system. We could have
multiple clients across the Internet building one or more modules each
one and reporting the build status to a central server. And what's more,
it will allow us also to distribute a debian package (or a tarball) with
a client tinderbox configuration for each module, this way, we could
have the maintainer for the compilation of the module X, another for the
module Y, etc.

We'll release more info next week, when we expect to have a more mature
solution. What do you think?

Regards.