[BuildStream] Bazelcon hackathon result - using BuildStream with RBE

[Jim MacArthur <jim macarthur codethink co uk>]

At the Bazel Hackathon in New York on 11th October, me and Chandan (with 
help from some other participants) managed to get BuildStream to run 
commands on Google's Remote Build Execution service. 
https://groups.google.com/forum/#!forum/rbe-alpha-customers gives a 
starting point for RBE. Our work was very much a hack and it's a while 
off being supported by BuildStream, but it's good to know it can work if 
needed.

RBE is intended for use with Bazel build systems, so uses the remote 
execution API (REAPI), which BuildStream also uses, so we expected these 
services would interoperate. There are however a few things we need to 
change in BuildSteam to make this work correctly:

1) We currently don't distinguish between remote caches for artifact 
storage and for remote execution objects - BuildStream sends remote 
execution objects to the normal remote artifact cache. Since 
BuildStream's artifact caches use a superset of the CAS storage 
protocol, it will break when it finds RBE's storage endpoint and finds 
features missing. We need to split this up so we send artifacts to 
artifact servers and remote execution objects to remote execution CAS. 
This should be relatively easy, but needs extra fields adding to 
project.yaml to specify the remote execution CAS endpoint.

2) Instance name typically needs to be specified, so we need to allow 
specifying that in YAML. Currently we just use a blank instance name and 
rely on the endpoint URL to determine which remote execution service we 
want to use.

3) TLS security won't work for RBE, so we need different authentication. 
We used OAuth2 in the hackathon, which is relatively easy to set up, but 
may draw in some extra dependencies.

4) We need to supply a docker image, or a link to one, to RBE to run our 
commands in. So far as I understand it, the input filesystem we supply 
will be mounted somewhere in that docker image. We could just use a very 
generic image, running buildbox inside it if we think we need the extra 
isolation. We got away without a docker image in the hackathon because 
the test RBE instance had an unusual configuration.

We should do (1) in any case. Whether we proceed with the rest of the 
work for RBE will depend on how much interest there is in using RBE with 
BuildStream. In any case, it's a useful confirmation that our use of the 
REAPI is correct.

Jim