Re: Dogfooding discussion

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Sorry missed that there was more replies...

[...]
> > This feels like adding "build" and "deploy" to a cycle which is
> > currently just "edit" and "test" (using the `pip3 install --user -e .`
> > approach).
> >
> > Doing a lot of work from within a `bst shell` open on a workspace
> > sounds a bit more likely...
>
> I'm not sure I get your meaning here.
>
> From my understanding we're intending
> that others should be able to use `bst shell` to debug and test,
> so if it's harder to do that than `pip3 install --user -e .`
> we've got some work ahead of us to make it easier.

No. These serve different purposes.

If you want to test BuildStream against a specific integration, or
target an appliance where BuildStream is specifically meant to run,
then `bst shell` would allow you to test BuildStream under real world
conditions, directly in the integrated system in which it will run
(ignoring possibly surmountable namespace issues which prevent us from
running BuildStream inside BuildStream).


> > > 3.  We use `bst shell` to test BuildStream.
> > >
> > >     Instead of having our container include all the dependencies
> > >     that are necessary to run buildstream.git's test suite,
> > >     we instead include those by defining them in .bst files.
> > >
> > >     We use `bst shell` for both running the test suite and test builds,
> > >     using `--mount` to bring the test project and your local caches in.
> >
> > But as I understand it, this is currently not possible (maybe it will
> > be possible with future kernel developments ?).
> >
> > At least there needs to be some research towards whether this is even
> > possible: as I understand it Flatpak cannot be run in a Flatpak, and as
> > we use the same container technology, I expect that BuildStream cannot
> > run a build launched from inside a `bst shell`.
>
> It works if either unprivileged user namespaces are permitted by your kernel
> or for the inner invokation of `bwrap` we use `--unshare-user`.
>
> We can emulate the kind of sandbox we use in `bst shell` by running:
>
>     bwrap --unshare-pid --die-with-parent --bind / / --unshare-net \
>           --unshare-uts --hostname buildstream --unshare-ipc --chdir / \
>           --proc /proc --tmpfs /tmp --dev /dev --dev-bind /dev/full /dev/full \
>           --dev-bind /dev/null /dev/null --dev-bind /dev/urandom /dev/urandom \
>           --dev-bind /dev/random /dev/random --dev-bind /dev/zero /dev/zero \
>           --remount-ro / --unshare-user --uid 0 --gid 0 /bin/sh
>
> Attempting the same command in the created sandbox resulted in:
>
>     bwrap: No permissions to creating new namespace, likely because the kernel does not allow 
> non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl 
> kernel.unprivileged_userns_clone=1'.
>
> It worked after dropping the `--unshare-user` from the command.

If we can do this in a reliable way, then yeah it would be nice to
build a Flatpak of BuildStream, using BuildStream, this would be the
right amount of dogfooding IMO.


> <snip>
> > > ---
> > >
> > > There's also a related argument that we should be publishing distro packages.
> > >
> > > This could take the place of using a container, but not immediately
> > > since the current container building approach involves installing from source.
> > >
> > > There are benefits and drawbacks of using a container
> > > which may be worth considering when packages exist as an alternative,
> > > but we can defer a decision on that to some degree,
> > > since we could incorporate packages into the container build
> > > by having a CI job between running the tests and building the container
> > > that publishes to a package repository,
> > > which the container can add as a package source to install from.
> >
> > I think I would prefer to have not ever gone into the territory of
> > recommending using BuildStream within a container; it may be practical
> > for people in some scenarios (like Docker in GitLab), but there should
> > not be a need for launching it from a container as a tool on your own
> > host.
>
> I agree it shouldn't be *needed* but it is convenient,
> so ideally we'd provide some level of support for the work-flow.
>
> We're suggesting we go with a container to start with
> since it's something we already know works
> so we can start dogfooding with less up-front effort.
>
> It doesn't preclude doing the work to have proper packages.

If you have a trigger which causes a Docker image to be generated with
each latest/stable version of BuildStream, this would be useful for the
users of `bst-here` yes.

I'm not sure what it does for dogfooding, unless we have a Docker image
plugin and a BuildStream project which we use to generate the image
using BuildStream.

We'd like to have a Docker image generating plugin... and it would be
helpful to `bst-here` users to automate the image generation as long as
we have to maintain `bst-here`.


> > > ---
> > >
> > > Since we'll need a base system capable of hosting BuildStream to use `bst shell`
> > > it's possible we could provide containers based on that instead of a distro.
> > >
> > > We could incorporate dogfooding of the freedesktop-sdk or Baserock,
> > > into our dogfooding story if we were to build our buildstream.bst on top.
> > >
> > > I suspect since we may end up needing to do most of this to support
> > > testing buildstream.git with `bst shell`,
> > > it may be a relatively small amount of extra effort to use it for our container.
> >
> > On a similar subject, I'd like to get coverage for `bst source-bundle`, 
> > it would be nice if we used the freedesktop-sdk project to assert that
> > it's actually useful for the purpose it was created for:
> >   o Run source bundle on the dependencies you need for a system with
> >     BuildStream
> >   o Build the cross built runtime for another arch, plus kernel
> >     and init script
> >   o Boot the cross built runtime in a VM, the init script launches
> >     the build script generated by `bst source-bundle`
> >   o Once that compiles successfully in it's vm...
> >   o Run the BuildStream test suite on that built in that emulator
> >
> > I think we're going to get dogfooding along the way, and in the right
> > places; I just don't think we should rally our efforts under the banner
> > of dogfooding.
>
> That sounds like too long a task to not automate,
> at which point we wouldn't get any feedback on the user experience,
> which we're hoping to improve by dogfooding
> by building and testing BuildStream in BuildStream.

It would not get us any more feedback on the user experience, but it
would cover a sorely empty gap in our test coverage - and it would be
done by using a BuildStream project that drives the whole thing and
needs to be rather fully featured to do so; meaning that we do tick the
dogfooding checkmark with it, and we even cover a fairly large feature
surface with it too (i.e. project needs to cross-bootstrap a new
runtime).

Cheers,
    -Tristan