BuildStream BoF at GUADEC: Wednesday session minutes

From: Agustín Benito Bethencourt <agustin benito codethink co uk>
To: buildstream-list gnome org
Subject: BuildStream BoF at GUADEC: Wednesday session minutes
Date: Tue, 17 Jul 2018 12:32:49 -0700

Hi,

please review the minutes of our Wednesday BoF

## Topics not covered during the BoFs

* Improvements in Code Review governance. - @tristanvb
* Gitlab related topics
* Notifications through mail - @toscalix
* Evaluation of the capabilities to integrate Gitlab and mailing lists
to send merge requests and diffs to the mailing list. - @toscalix
* BuildStream release
* Schedule - @toscalix
* Goals - @tristanvb @jjardon and @toscalix
* Key features - @tristanvb
* Release checklist - @toscalix

# Wednesday session

* SourceCache - Richard Maw
* Summary of yesterday's Flatpak BoF - Valentin
* Dogfooding BuildStream - Richard Maw
* Freedesktop SDK - Javier and Adam
* Bootable images (complete systems) - Javier
* VCS Reduction - Richard Maw
* Conference evaluation - All

## SourceCache

* We reuse sources in various cases and could optimise the staging
* Splitting tasks: CAS cache and xxx
* Currently believe we know the requirements from experience with
BuildGrid - also uses a CAS cache
* Will be extra work to reformat data, but hopefully it will pay off for
other workers
* Need a cache ID for sources
* may already have one: Source.get_unique_key()

## Summary of yesterdays Flatpak BoF

* Valentin summarised the conclusions of the BoF and the conversations
with GNOME devs, specially Alex Larson.
* Discussion about how to move forward
* For each module there is an element
* Each Flatpak package need 2 elements:
* Compose to split and run integration commands
* Flatpak image that takes the /app sub-tree out of the artifact to
package a Flatpak app,
* Because we need at least the runtime plus the .Debug extension, we
need 4 elements to package each application.
* Re-use what's in compose - make something that can work with compose.
* Generate different directories per domain of the split rules, still have
an artifact at the end, check that out, run Flatpak build-export on the
directories
* Things to improve on freedesktop sdk:
* Clean up of junctions - we have two - One for config, one for sdk
itself. Simplify by using just one junction.

https://gitlab.com/valentindavid/org.gnu.Hello/tree/valentindavid/supertuxkart
(b515a232db73d5aca1f10e5703877aea13e4431c)
https://gitlab.com/BuildStream/buildstream/tree/valentindavid/331_include
(6028168de6fc4bbf27b7e04c96a336d641536dc5)
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/tree/valentindavid/conf-includes-2

## Dogfooding

* We have a problem that developers are making changes to buildstream
without using BuildStream so the only feedback we get is from users.We can
advance some of those issues by using the tool ourselves.
* there's been debate about best way to do this
* we could build a system to host buildstream in and use bst shell to
run the test suit in this
* we could build other images as well (Flatpak/container) images out
of this so that we can use the current version
* There's worries that we could end up over optimizing for our own use
case
* Potential blocker - Linux kernel, before 4.18, does not support
unprivileged fuse mounting - you can't combine usernamespace and FUSE, and we
need FUSE.
* So one option is just to wait for 4.18 which is released in a couple of
weeks
* Valentine emphasises the point that the bottom line is simple: Our
developers aren't using the tool and that using the tool in anyway is going to
be a big step forward.
* It looks like we have users working with slightly different workflows:
bst shell vs workspaces.
* Agustin has points he wants developers to consider in order to ensure we
do not go too far.
* dogfooding is about testing a tool as it is consumed by users and
use it as users do, mimetising their workflow as much as possible. It is
advancing problems they will have.
* Dogfooding vs simulation/emulation.
* we are testing as users *are going to* use the tool, not how they
are using it at the moment. There is a risk of getting an experience that is
not real.
* questions that this is actually dog fooding but still sees value in
doing it
* Problem is perhaps that bst is too different from the stuff being
developed by downstream users
* Let's ensure we don't set up an imagined workflow here - need to base
it on actual use cases.
* AB: simulate the dev workflow in one example and then set up test to
run over and over again - that is a simulation / emulation, rather than dog-
fooding
* We can test the workflow of developing using workspaces, but may not be
able to test bst shell due to the above issues
* We could make an example building buildstream, BuildGrid and buildbox
* Tristan M points out that python developer workflow means we'll never
actually build anything using buildstream for buildstream
* we could use buildstream to build packages from which we could install
buildstream.
* Javier: what about create a Flatpak? We will partially resolve the
distribution problem and exercise the Flatpak-image plugin that we want to
improve anyway

Summary:
* Clear path forward is to have a project that we can open a workspace
with, develop in that way, build a package that we can install.
* We may not get shell working because of kernel versions but it's
something to look at. Building Buildbox is a good target.
* So, outcome should be: project with bst files for 3 components to build -
BuildGrid, buildbox, buildstream.

## Freedesktop SDK

* Waiting for features
* Marked in issue tracker as BuildStream
* Generate manifests etc.
* Release - As soon as blockers are fixed. A matter of weeks.
* It would be nice if BuildStream release a tag, so Freedesktop SDK can
use that instead of a SHA1.
* They want to change the release numbering to release date based
* Want to replace semantic versioning by date based. Older dated
releases will end up with more updates.
* Will be Ubuntu-like 18.07.0 with bugfixes will be 18.07.1
* At least annual releases.
* We may end up with a year with no ABI changes which would mean not
making a yearly release.
* We think it's unlikely since openssl changes ABI, but we might
stick with an older openssl version for the sake of applications even if there
is a newer openssl version.
* at present openssh isn't compatible with later versions of
openssl, so updating right now is a challenge
* We can also have multiple versions of libraries installed in
parallel. (should raise this with Flatpak guys?)
* There's a messaging problem in the guarantee of how often there will
be a release.
* It's too young a project to need to have a regular release cycle,
but it may make it more attractive as a platform.
* For Open Source applications it would be possible for FlatHub to try
building applications against newer releases, but proprietary applications
can't do that.
* Currently thinking that every 2 years would be a good cadence.
* We are not sure on how long we will port security patches to older
versions
* Not every GNOME component makes a release with GNOME's release
cadence, e.g. if there are no interesting changes.

Release Candace:

* Aim for release once every year, unless there has been no ABI breakage,
then no need to release. OR if there's a big application or feature change, we
should release.
* Message here to advance is that the release is based on ABI breakage,
rather than an arbitrary date.
* However, GCC could be a spanner in the works here...if you want to stay
up to date with the latest GCC version, you need to release every year, as GCC
does. So are we saying that we need to release every year? To be decided.
* ACTION: Javier to come up with two release proposals - ask for the other
stakeholders' input here.

Options discussed:
1. We release every year, _only_ if:
2. There is a ABI break
2. There is a major release in a component
1. We release stable,
2. _only_ gets security updates
2. All updates go to master now
1. We explore changing release schedule for Platform and SDK

Javier's Proposal draft:
- We release stable, _only_ gets:
- security updates
- stable releases (tested carefully), no ABI breaks / API build breaks.
- We release every year, _only_ if:
- there is a ABI break
- there is a "API build" break (apps might not compile because new major
releases of important packages (GCC))
- Looking at the GCC cadence, this is going to happen

- We only maintain current release and previous one, this means:
- stable release gets 2 years of security updates
- We maintain 3 release at any given time:
- development
- stable
- stable-1

## Bootable images

* MR already available and working (in the CI, booting a system)
* Only remaining a Dependency problem:
* The freedesktop-sdk runtime shouldn't depend on systemd, but
systemd is needed for booting.
* d-bus builds differently based on whether it depends on systemd or
not. Without systemd it is usable as a Flatpak container runtime. With systemd
it is usable as an image.
* Can have a conditional dependency on systemd if we want a variant of
freedesktop-sdk that can run as an image
* ACTION: Javier, valentin and tristan to explore bst conditionals to fix
the already existing MR

## VCS Reduction

* Juerg and Richard had a go at shell scripts to solve this on Monday
* https://gitlab.com/BuildStream/buildstream/snippets/1731497
* git stability disagreements between Juerg and Richard
* TVB: tested with Juerg's shell script overnight, everything we build
from git has built, inspected one of the projects that uses git describe, it
has the correct version number
* Need to do more thorough testing though rather than just a manual check
of one.
* There's a risk here that the shell script could be an ugly
implementation that breaks things later - but the impact is not huge.
* Conclusion: this solution is not the most optimal, there is risk
attached, there is no such thing as a perfect solution here - this is actually
a feature that's not supported in git.
* We would need to be able to pre-process a git history such that a
subsequent call to `git describe` will work with any provided options.
* AB: so as part of our release we have an page with 'known issues' - this
is completely normal and we can deal with that.

Summary to approach: history which doesn't include objects, includes the tag
in the cache key, though possibly not in the tracking data, as we want to
ideally avoid a false hit if that tag has moved since history was updated
(this would clearly be undesirable since we then cannot guarantee
reproducibility. Cache misses may be expensive, but a false cache hit is
scary).
We should create a prototype / demo / first draft - before trying to create
perfect.

## Documentation

* Currently focused on newcomers, we can do more, including diagrams. We
have demands for this which is a positive sign.
* For technical documentation, we have an established process in terms of
how to write this.
* Not currently got a clear description of what BuildStream even is.
* The topic of how it should be documented comes up every time we spawn a
new sub-project. If we think documentation of BuildStream is doing well we
should use the same process for sub-projects.
* We should come up with lists of what examples and other documentation we
think we should have, so when developers have time, can pick those
documentation tasks up.
* We should have a clear structure to follow for any newcomers to follow
in order to submit and contribute docs.
* Phil says we could add a paragraph in HACKING.rst to cover this - TVB
says that this policy is covered on gitlab.
* This is an area we can improve. Make it easy for people to contribute.
* ACTION: tristan to check and put in HACKING the guide he shared with
Alberto; DONE
* Could have an edit button on the docs where it takes you directly to the
.rst on gitlab and you can amend it and submit an MR.
* AB: Let's keep consistency between all of the projects under the
BuildStream Umbrella.
* Analyze use cases and write doc specific for them
* Developer
* Integrator
* Flatpak packager
* ...
* One one to tackle this is to write examples for each case
* We are missing documentation for examples, specifically:
* I want to put a cmake project in BuildStream
* I depend on project A, but i want to over-ride how it builds a
library, because i don't want to build it twice. JJ raised a ticket for
this...
* So the two examples that live in buildstream-examples make use of bst-
external plugins. We should migrate these examples to bst-external
documentation, i.e. how do we write and use plugins that aren't supported by
buildstream core
* We can then link to this into the main repo.
* For related projects (e.g. bst-external) it should have its own
documentation, and they should link to each other and have as close look and
feel as each other as possible, so the transition between documentation for
one project and another is seamless apart from the URL pointing to a different
project's documentation now.

* Important pre-release docs
* overides
* conditionals
* junctions
* filters
* workspaces

## Conference evaluation

No minutes from this topic.
* Each participant in the BoF went through the highlights and points to
improve about GUADEC and the BuildStream specific activities or events.

## Participants on the Wednesday session

* Jürg Billeter - juerg billeter codethink co uk
* Agustin Benito Bethencourt - agustin benito codethink co uk
* Richard Maw - richard maw codethink co uk
* James Ennis - james ennis codethink co uk
* Phil Dawson - phil dawson codethink co uk
* Thomas Coldrick - thomas coldrick codethink co uk
* Finn Ball - finn ball codethink co uk
* Valentin David - valentin david codethink co uk
* Laurence Urhegyi - laurence urhegyi codethink co uk
* Javier
* Adam
* Tristan

################################

--
Agustín Benito Bethencourt
Principal Consultant
Codethink Ltd
We respect your privacy. See https://www.codethink.co.uk/privacy.html

[Date Prev][Date Next] [Thread Prev][Thread Next] [Thread Index] [Date Index] [Author Index]