Re: GNOME Build situation and BuildStream

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Matthias,

I realize now that this was too much information at once (even for the
involved reader as opposed to a fly-by reader).

So I'd like to thank you for your mind share.

On Wed, 2017-04-26 at 16:39 -0400, Matthias Clasen wrote:
> Tristan,
>
> again, it is impossible to reply to an email of this length. I can
> only give a few general comments, beyond that, we really need to sit
> down face-to-face and discuss this. I hope you are going to be at
> Guadec ?

I will certainly be around all week at GUADEC to meet with you and
anyone who wants to discuss :)

I am preparing a talk on this subject, but perhaps I should also try to
organize something more hands on, maybe a BoF or such would be good.

> My general comments:
>
> What you are describing here (and in your previous communications)
> looks like a big, all-encompassing system, with lots of its own
> terminology and a complete worldview of how things should be built. I
> prefer a system that starts small and solves one problem initially,
> and then maybe grows over time.

I can see how it can come across this way, we are trying to break the
trend of having a build system as something that is tied to any
particular deployment/use case.

As such, I needed to give consideration to a lot of use cases to be
sure that this is something that fits, and is also an improvement over
what exists. These considerations will be reflected in my
communications and I can see how one might think this appears to be
some kind of huge monolith which does everything.

However this is exactly the opposite of what I'm trying to achieve,
instead we are striving to "do one thing well" and are making an effort
to ensure we're doing it the right way, for any use case.

So, the core codebase itself should remain small over time, with really
the sole purpose of being:

   "A format and engine for modeling and executing pipelines of 
    elements which perform mutations on filesystem data within an
    isolated sandboxed environment"

In time, I expect that an ecosystem of plugins and projects will grow
around this, and use cases I had not even foreseen will come to light.
This has already started to happen in some ways, as Jochen Breuer's
commented on my blog here:

   https://blogs.gnome.org/tvb/2017/02/06/introducing-buildstream-2/

And as a result has started to work on a plugin which would allow
importing distro packages and building on top of those bases:

   https://gitlab.com/BuildStream/buildstream/issues/10

> The system you describe seems to be all about centralization, and
> about introducing a new language to describe what we build. That is
> by-and-large what we already have in various incarnations that you
> describe: jhbuild modulesets, the continuous manifest, flatpak
> runtimes. I can get behind the idea of unifying these into a single
> way of describing a multi-module build.
>
> But I've also seen things mentioned like 'conversion scripts for
> flatpak'. And I think that is exactly the opposite of what we need
> for application building.

I may be mistaken, but I have a feeling you are getting the same
impression which Christian had last month, which I tried to explain in
this email:

   https://mail.gnome.org/archives/desktop-devel-list/2017-March/msg00003.html

> If we want to convince 3rd party applications to use flatpak, we
> can't treat it as an intermediate format that we generate from some
> other source, and just use somewhere in a centralized build process.
> We need to embrace it ourselves and use it, just like we expect 3rd
> party applications to use it.

So at the risk of being repetitive, I am completely behind application
authors maintaining their own build metadata themselves, building
flatpaks themselves and/or submitting build metadata to a "flathub" to
have them built and published to users.

Of course this makes sense, because the application authors themselves
are usually best situated to know what should be in their bundling
build metadata.

So let me try to break down how I would see this work (I realize,
already a long email):


  GNOME core modules and services (excluding Flatpak apps)
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  These can be expressed in a single format/repository for all the
  interesting purposes:

    o Performing CI
    o Creating bootable GNOME images on top of some base system,
      mostly for developers
    o Release modulesets
    o Producing the GNOME Flatpak runtime and SDK

  This is of course, one centralization.


  Flatpak Applications
  ~~~~~~~~~~~~~~~~~~~~
  Considering the benefits which the GNOME core modules and services
  get by representing build metadata for multiple purposes, a given
  application developer team may also benefit in similar ways.

  This is without centralizing everything into one big build blob, or
  using intermediate formats or anything like this.

  Now, this may be more of a personal goal, a bit more ambitious,
  maybe best punted to later on, but consider that for instance: many
  applications still today have OS specific subdirectories where build
  metadata in various formats are used to create application bundles
  for OSX and win32.

  Also, I'm not exactly rooting for "snap" here, but it may very well
  be that an application developer who wants to deploy a Flatpak
  bundle, may also want to deploy a "snap".


Now, even if the arguments above for using BuildStream *instead* of
flatpak-builder to build apps never bear fruit (maybe win32 bundles are
so immensely different that there is little or no gain to representing
those builds in the same tree ?), it would still be a net win to
maintain one less build system (and this is where conversion scripts
for me become interesting, as an easy migration path, not an
"intermediate format").

I realize that Alex has already made a statement that flatpak-builder
will continue to be supported and maintained, possibly indefinitely,
but I wont lie about this: I really want to save developer hours.

If it is possible that we can reuse the same body of work to build many
things, and the result is a reduced maintenance burden on the Flatpak
developers, I would much prefer to see valuable developer hours focused
on how Flatpak actually works, than diverted into menial tasks like
getting builds to work.


I hope this response clears up some things for you.

And thanks again for taking the time to reply and sharing your
considerations, despite the huge size of my initial email :)

Best Regards,
    -Tristan