Re: Build sheriffs for GNOME

[Emmanuele Bassi <ebassi gmail com>]

Hi;

On 21 January 2016 at 15:22, SAHIL SAREEN <sahil sareen hotmail com> wrote:

> I feel we should spend time building a simple "build sheriff" (maybe an extension of 
> https://build.gnome.org) that auto-files a "cri" bug for the module whose build failed. This should make 
> sure that it isn't creating duplicate bugs(reopening the old one if exists) for the same module and append 
> a comment giving the link to the respective build failure.
>
> The package maintainers(or the #testable s) should then either revert the bad commits or fix them and close 
> the bug.
>
> Thoughts?

There are two issues with that, and they cannot be solved via automation:

  1. since Continuous does not do reverse dependencies when building,
the component that may be breaking the build may not be the actual
component that needs a tag or a revert. If, say, component X
introduces an API change, then component X will likely build fine, but
every other component that depends on component X and uses that API
will fail to build. We cannot send an email/notify on IRC/open a bug
for other components.

  2. the goal is not to notify maintainers; the notification is
actually an anti-goal. The goal, here, is to ensure that the maximum
time for a build breakage is around an hour. The goal is to ensure
that the *whole* of GNOME *always* builds at any given time of day.

We notify maintainers today, by filing bugs; by joining IRC channels;
by sending emails. That can take ages to get a response because
maintainers are humans, and have other things to do. I filed a bug and
tagged a component of the manifest because of a broken build (caused
by one of its dependencies) mid-December, and the maintainer was on
vacation. This means that the components in question were not built
for three weeks because of the tag. A revert would have avoided the
tag, and kept the world building.

Bugzilla can deal with duplicate bugs; the amount of effort spent
there is negligible.

Ciao,
 Emmanuele.

> From: desktop-devel-list <desktop-devel-list-bounces gnome org> on behalf of Emmanuele Bassi <ebassi gmail 
> com>
> Sent: Thursday, January 21, 2016 8:24 PM
> To: Desktop Development List
> Subject: Build sheriffs for GNOME
>
> Hi all;
>
> Many of you know about GNOME Continuous, and build.gnome.org — and for
> those who don't, here's two handy links:
>
>   - https://build.gnome.org
>   - https://wiki.gnome.org/Projects/GnomeContinuous
>
> In short: we're currently building the core GNOME and some
> applications every time something gets committed to git.gnome.org;
> we're also doing various tests — like smoketesting the session and
> running applications — and building VM images out of the build
> process.
>
> This effort led to various benefits, including JHBuild not constantly
> being in a broken state, and most projects hosted on git.gnome.org
> finally building when builddir != srcdir, which should improve the
> ability of maintainers to do `make distcheck` on release days.
>
> What we need now, though, are "build sheriffs", i.e. people that watch
> the build process and ensure that it continues smoothly. Ideally,
> every maintainer would be on IRC, and would join the #testable
> channel; sadly, that's not the case. Many are not even on
> #gnome-hackers, which means they miss the notification that the
> something broke the build. We cannot always send emails to the last
> committer of a broken module because GNOME is a complex project, and a
> change in a dependency may indeed break your project, even if you
> didn't know about it.
>
> For the past few months a few people, myself included, have been
> hunting down build breakages; if we exclude the issues with the build
> machine itself throwing a fit — something that usually gets fixed by
> Colin kicking it — the vast majority of build breakages come from
> issues inside GNOME projects.
>
> What usually happens when a build goes into perma-red (i.e. it keeps
> failing over the same component) is that somebody on the #testable IRC
> (usually me or Colin Walters) tags the module inside the manifest,
> opens a bug, and hopes that a fix get applied and  on the channel so
> that the tag gets reverted.
>
> This is not enough, and it does not raise the bar in keeping GNOME in
> a buildable state. It actually lowers it a fair , to the effective
> point that *nobody* cares about Continuous builds.
>
> I want this to change. I want to be able to revert failing commits on
> the offending modules, if they are hosted on GNOME infrastructure, if
> they fail for more than N hours, and *then* open a bug about it.
> Ideally, I want to tag only modules that are *not* hosted on GNOME
> infrastructure, as they are beyond our control and commit
> capabilities. In short, I want to ensure that GNOME maintainers become
> a bit more proactive in giving a crap about their modules breaking on
> something that is not their own computers.
>
> So, here's what will happen in the near future in case of a build break:
>
>   - a build sheriff will identify the component that is actually
> breaking; this usually takes about a minute of reading the build log
> linked by the IRC bot
>   - the build sheriff will try to reach the maintainer of the
> component and see if they are on IRC or any other form of IM; I
> strongly suggest you either use the same nickname as your GNOME user
> id in the DOAP file of your projects, or we'll need to figure out a
> way to link your IRC nickname with the project description
>   - if nothing happens for a reasonable amount of time (I'd give one
> to three hours)
>     - if the offending component is not hosted on git.gnome.org, the
> build sheriff will tag the component in the gnome-continuous manifest
> file
>     - if the offending component is hosted on git.gnome.org, the build
> sheriff will revert the commits that caused the breakage
>   - in either case, a bug will be filed in the bug tracker for the
> component, if one is not already open (that's why you should always
> list bugs into the commit message)
>
> What does a build sheriff need to know? Usually, breakages involve the
> build system. If you know autotools and/or CMake enough to fix issues
> with them, you should consider joining. The wiki page on GNOME
> Continuous describes the system — so you should take some time to
> familiarise yourself with it:
>
>     https://git.gnome.org/browse/gnome-continuous
>
> Especially the manifest file.
>
> The process of keeping GNOME building will be much more smooth and
> painless the more people we have watching the Continuous build on
> #testable. Ideally, you should *always* look at
> https://build.gnome.org or join the #testable IRC channel.
>
> For the time being, Colin Walters, Javier Jardón, and myself will be
> watching the build as much as we can spare (we're also busy with our
> own jobs ;-)) and try to fix issues; any help is much appreciated,
> though.
>
> Ciao,
>  Emmanuele.
>
> --
> https://www.bassi.io
> [ ] ebassi [ gmail com]
> _______________________________________________
> desktop-devel-list mailing list
> desktop-devel-list gnome org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list



-- 
https://www.bassi.io
[ ] ebassi [ gmail com]