[BuildStream] BuildStream Gathering 2018 topic: Bazel and BuildStream (and RE)

[Chandan Singh <chandan chandansingh net>]

Hi,

I had proposed the "Bazel and BuildStream" topic, here are some details and
motivations for it. While writing about this, I have also added RE, i.e.
Remote Execution to the topic as it is somewhat related.

Bazel and BuildStream and RE
============================

1. Bazel in BuildStream
-----------------------

First, we would like to add a Bazel [0] plugin to BuildStream. This plugin
would be useful for integrating projects built with Bazel with the rest of the
world in a reliable way. In general, this plugin would work by dynamically
creating a Bazel WORKSPACE file [1] (not to be confused with BuildStream
workspaces).

There are a few interesting caveats to consider though:

- The Bazel binary is roughly 160 MB in size so downloading that for running
  integration tests might make them quite slow. Also, Bazel does not run on
  Alpine currently. Since we are currently using Alpine as base platform for
  our integration tests, switching to something like Debian will make the tests
  even slower. So, we need to think about how to run tests for this plugin.

- We also need to think about how to generate this WORKSPACE file as
  naively declaring a dependency on all the files provided by all the build
  dependencies will work but will not be very performant.

- BuildStream will have to provide a way for element plugins to have a cache
  directory. This will be needed because Bazel has maintains a cache of the
  dependency graph and its performance would degrade if it has to calculate
  that graph on each run.

- Bazel likes to have source-to-source dependencies and needs to be able to see
  the sources for its dependencies for its caching to work properly
across elements.
  So, this plugin should stage sources for all its build dependencies
  rather than their output.

2. BuildStream and Remote Execution
-----------------------------------

BuildStream has recently started supporting the Remote Execution API. While
this does work with BuildGrid, it would be nice to have it work with RBE [2] as
well. RBE does follow the same RemoteExecution API but we need to figure out
how to handle authentication flow with RBE and how to use CAS. Additionally,
we will also need to figure out how to run jobs inside Docker as we essentially
want to mount a custom root filesystem inside the container, which is not quite
supported.

Jim, Juerg and I started looking into at the recent BazelCon Hackathon but it
would be good to properly capture all the issues and come with an action plan.

Looking forward to seeing you all at the Conclave ^W Gathering :)

[0]: https://bazel.build/
[1]: https://docs.bazel.build/versions/master/be/workspace.html
[2]: https://docs.bazel.build/versions/master/remote-execution.html

Cheers,
Chandan