[BuildStream] Sandboxing backends and platforms: Drop chroot in favor of buildbox-run

[Jürg Billeter <j bitron ch>]

Hi,

TL;DR: We now have the platform-agnostic buildbox-run sandboxing
backend. Let's drop the chroot sandboxing backend.

Background
~~~~~~~~~~

We've recently merged the buildbox-run sandboxing backend. The
BuildStream support code is platform-agnostic and should work on any
system that has a buildbox-run implementation.

It's still considered experimental as it's missing a few features¹
(mainly in the area of interactive shell support) and is not enabled by
default. However, it is tested in CI with buildbox-run-bubblewrap
(using FUSE) and buildbox-run-userchroot (using hard links). Both pass
the test suite with a few expected failures marked as xfail.

Both CI jobs are running on Linux as we don't currently have any Unix
CI runners, however, buildbox-run-userchroot doesn't have any Linux
dependencies and should work on other POSIX/Unix systems as well.

Proposal
~~~~~~~~
I'm proposing to drop the chroot sandboxing backend (and the 'unix' CI
job) in favor of buildbox-run-userchroot and other buildbox-run
implementations. The purpose of the chroot sandbox was to have a
backend that works on most POSIX systems as the default bubblewrap
sandbox is Linux-specific.

However, the chroot sandbox has various issues, the main one being that
it uses Linux-specific mount flags (bind mounts) and relies on FUSE for
hard link protection and thus, it doesn't actually work on non-Linux
systems without additional patches². Also, it requires the user to run
BuildStream as root, which is not something we generally want to
recommend.

Setting up buildbox-run-userchroot is not trivial as it requires two
user accounts and userchroot is restrictive about permissions. However,
two user accounts are required to protect the local CAS cache from
corruption when using hard links without platform-specific features
such as a FUSE layer (or bind mounts or reflinks).

I'd like to encourage users on non-Linux platforms to test and
contribute documentation how to set up BuildStream with buildbox-run.
It would be great to have CI jobs for other platforms. Additional
buildbox-run implementations are also welcome (e.g., for FreeBSD
jails).

Please note that the recommended buildbox-run implementation for Linux
is buildbox-run-bubblewrap with buidlbox-fuse, which is simple to set
up, if the necessary dependencies are available.

Am I missing a use case for the chroot sandboxing backend that is not
covered by buildbox-run? Or are there any other concerns?

Cheers,
Jürg

¹ https://gitlab.com/BuildStream/buildstream/issues/719
² https://gitlab.com/BuildStream/buildstream/issues/190