Re: [BuildStream] Bazelcon hackathon result - using BuildStream with RBE

[Martin Blanchard <martin blanchard codethink co uk>]

Hi Jim,

On 19/10/18 15:10, Jim MacArthur via BuildStream-list wrote:
> At the Bazel Hackathon in New York on 11th October, me and Chandan (with 
> help from some other participants) managed to get BuildStream to run 
> commands on Google's Remote Build Execution service. 
> https://groups.google.com/forum/#!forum/rbe-alpha-customers gives a 
> starting point for RBE. Our work was very much a hack and it's a while 
> off being supported by BuildStream, but it's good to know it can work if 
> needed.
>
> RBE is intended for use with Bazel build systems, so uses the remote 
> execution API (REAPI), which BuildStream also uses, so we expected these 
> services would interoperate. There are however a few things we need to 
> change in BuildSteam to make this work correctly:
>
> 1) We currently don't distinguish between remote caches for artifact 
> storage and for remote execution objects - BuildStream sends remote 
> execution objects to the normal remote artifact cache. Since 
> BuildStream's artifact caches use a superset of the CAS storage 
> protocol, it will break when it finds RBE's storage endpoint and finds 
> features missing. We need to split this up so we send artifacts to 
> artifact servers and remote execution objects to remote execution CAS. 
> This should be relatively easy, but needs extra fields adding to 
> project.yaml to specify the remote execution CAS endpoint.
>
> 2) Instance name typically needs to be specified, so we need to allow 
> specifying that in YAML. Currently we just use a blank instance name and 
> rely on the endpoint URL to determine which remote execution service we 
> want to use.
>
> 3) TLS security won't work for RBE, so we need different authentication. 
> We used OAuth2 in the hackathon, which is relatively easy to set up, but 
> may draw in some extra dependencies.
>
> 4) We need to supply a docker image, or a link to one, to RBE to run our 
> commands in. So far as I understand it, the input filesystem we supply 
> will be mounted somewhere in that docker image. We could just use a very 
> generic image, running buildbox inside it if we think we need the extra 
> isolation. We got away without a docker image in the hackathon because 
> the test RBE instance had an unusual configuration.
>
> We should do (1) in any case. Whether we proceed with the rest of the 
> work for RBE will depend on how much interest there is in using RBE with 
> BuildStream. In any case, it's a useful confirmation that our use of the 
> REAPI is correct.

Thanks for sharing this! There also is an opened issue for (2) here:

    https://gitlab.com/BuildStream/buildstream/issues/627

> Jim