[BuildStream] BuildStream and BuildBox progress and plans

[Jürg Billeter <j bitron ch>]

Hi all,

We're making progress on various BuildBox components and are also
working towards using them in BuildStream. This mail is intended to
provide a status update and a starting point for discussing how to move
forward to complete the integration.

There are two mostly independent aspects to this but I'll cover both
for a more comprehensive overview:

* Use buildbox-casd for CAS access
* Use buildbox-run for local sandboxing


# Use buildbox-casd for CAS access

Tracking issue: https://gitlab.com/BuildStream/buildstream/issues/1042
Initial MR: https://gitlab.com/BuildStream/buildstream/merge_requests/1499

This is about using buildbox-casd for write access to the local cache,
expiry handling, and downloading from and uploading to remote CAS
servers.

Initial code for this is nearly ready. I expect to take MR !1499 out of
WIP either this week or early next week. This should provide a
significantly improved user experience for local cache expiry and allow
sharing of remote CAS connections among jobs, i.e., this is expected to
resolve #734 and #810.

Planned follow-up MRs:
* Add communication channel between bst and casd to restore displaying
'Cache Usage' status in frontend
* Use casd to check availability of remote CAS endpoints
* bst-artifact-server: Forward requests to casd instead of using our
own Python implementation
* Batch CAS writes to reduce round trips between bst and casd
* Use FetchTree/UploadTree to reduce overhead for downloading/uploading
complete directories


# Use buildbox-run for local sandboxing

Tracking issue: https://gitlab.com/BuildStream/buildstream/issues/719
Initial MR: https://gitlab.com/BuildStream/buildstream/merge_requests/951

This is about replacing all local sandboxing backends with buildbox-run 
to share code for local and remote builds for platform-specific
sandboxing mechanisms (e.g., bubblewrap and userchroot) and staging
mechanisms (e.g., hardlinking and FUSE).

Completely optional support for testing the buildbox sandboxing backend
has recently been merged and this is also covered by a new CI job. This
is a good first step, however, this still uses the standalone buildbox-
fuse instead of buildbox-run and more work is required to completely
replace BuildStream's current bubblewrap (and chroot) sandboxing
backend.

Planned follow-up MRs in BuildStream:
* Support workspace builds by revamping workspace handling to use CAS,
see #985. This will also be required to support workspace builds with
remote execution.
* Document installation of buildbox-run and buildbox-fuse as
dependencies.

Points requiring further discussion:
* Hardlink protection for systems not supporting FUSE, replacing
SafeHardlinks. The plan is to run casd (or the sandbox) as a separate
user. However, details still need to be discussed to provide the best
user experience on the various systems we want to support.
* Consider potential issues due to CAS not storing timestamps. This is
also relevant to remote execution. It's also related to workspace
support as incremental builds will require timestamp support.
* Support read-only rootfs in buildbox or move BuildStream away from
read-only rootfs. This is also relevant to remote execution.
See https://gitlab.com/BuildStream/buildbox/issues/23

Required changes in BuildBox:
* Use buildbox-casd for CAS access in buildbox-fuse
  https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/issues/32
* Move sandboxing code from buildbox-fuse to buildbox-run-bubblewrap
  https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/issues/33
  https://gitlab.com/BuildGrid/buildbox/buildbox-casd/issues/8
* Support interactive mode in buildbox-run for bst shell
  https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/issues/27
* Support network access in buildbox-run for bst shell
  https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/issues/25
* Make buildbox-run-bubblewrap usable without user namespaces
  https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/issues/26


Any questions, suggestions or other comments?

Cheers,
Jürg