Re: [BuildStream] Bazelcon hackathon result - using BuildStream with RBE

[Ed Baunton <edbaunton gmail com>]

Thank you for sharing this Jim, this is very interesting indeed.

One point that I think wasn't mentioned here but was discussed during BazelCon was the proposed formalisation of the syntax to specify the platform. Under RBE one has to specify a Docker container and I believe there was the thought to having that syntax standardised so that workers can implement the protocol in a reusable standardised way: we should bring this up in the Remote Execution working group which is scheduled for the 23rd. I shall start putting together some agenda items for that meeting.

With regards to "TLS security won't work for RBE": the BuildGrid community is already having discussions on how to flexibly support different types of authentication; I believe there is a mailing list post somewhere there in the pipeline. It might make sense to move that discussion to that list. On the BuildGrid side that authentication will probably be pluggable since it is required to properly implement scheduling and prioritisation: you must be sure of *who* someone is before you prioritise their job. Depending on the complexity and number of dependencies it might make sense to centralise everything around OAuth2 out of the box.

recc already has support for authentication via the same APIs you mentioned: https://gitlab.com/bloomberg/recc/merge_requests/32

Ed

On Fri, 19 Oct 2018 at 10:10 Jim MacArthur via BuildStream-list <buildstream-list gnome org> wrote:

At the Bazel Hackathon in New York on 11th October, me and Chandan (with
help from some other participants) managed to get BuildStream to run
commands on Google's Remote Build Execution service.
https://groups.google.com/forum/#!forum/rbe-alpha-customers gives a
starting point for RBE. Our work was very much a hack and it's a while
off being supported by BuildStream, but it's good to know it can work if
needed.

RBE is intended for use with Bazel build systems, so uses the remote
execution API (REAPI), which BuildStream also uses, so we expected these
services would interoperate. There are however a few things we need to
change in BuildSteam to make this work correctly:

1) We currently don't distinguish between remote caches for artifact
storage and for remote execution objects - BuildStream sends remote
execution objects to the normal remote artifact cache. Since
BuildStream's artifact caches use a superset of the CAS storage
protocol, it will break when it finds RBE's storage endpoint and finds
features missing. We need to split this up so we send artifacts to
artifact servers and remote execution objects to remote execution CAS.
This should be relatively easy, but needs extra fields adding to
project.yaml to specify the remote execution CAS endpoint.

2) Instance name typically needs to be specified, so we need to allow
specifying that in YAML. Currently we just use a blank instance name and
rely on the endpoint URL to determine which remote execution service we
want to use.

3) TLS security won't work for RBE, so we need different authentication.
We used OAuth2 in the hackathon, which is relatively easy to set up, but
may draw in some extra dependencies.

4) We need to supply a docker image, or a link to one, to RBE to run our
commands in. So far as I understand it, the input filesystem we supply
will be mounted somewhere in that docker image. We could just use a very
generic image, running buildbox inside it if we think we need the extra
isolation. We got away without a docker image in the hackathon because
the test RBE instance had an unusual configuration.

We should do (1) in any case. Whether we proceed with the rest of the
work for RBE will depend on how much interest there is in using RBE with
BuildStream. In any case, it's a useful confirmation that our use of the
REAPI is correct.

Jim

_______________________________________________
BuildStream-list mailing list
BuildStream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list