Re: [BuildStream] BuildGrid and BuildStream 1.4 and API concerns

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Sander,

On Fri, 2018-11-30 at 16:51 -0500, Sander Striker wrote:
> Hi,
>
> > On Mon, Nov 26, 2018 at 10:02 AM Tristan Van Berkom via BuildStream-list <buildstream-list gnome org> 
> > wrote:
> > Hi all,
> >
> > BuildStream 1.4 will be coming out with support for remote execution,
> > which in a way means optional dependency on BuildGrid.
>
> No.  It is an optional dependency of services that implement the
> version of the Remote Execution API that are compatible with bst
> client side implementation.  As the protocol is under active
> development, if this is a concern, then let's mark the feature as
> experimental.

It is our responsibility to ensure that upgrades don't leave users in
situations where software does not work, so we should always keep in
mind that it is of course, always a concern.
 
> > My question to the developers involved in BuildGrid is:
> >
> >   What is the status of API stability of BuildGrid regarding the
> >   protocols and any parts which BuildStream will directly be
> >   interfacing with ?
>
> None.  BuildGrid currently makes no guarantees of stability other
> than tracking the latest version of the protocol while not having
> done a release with a non-zero major version number.  I envision this
> to be the case until we are comfortable the protocol and project
> don't undergo any major changes.
>
> In practice this will probably not be a problem.

What is your concern here ?

Are you worried that it will be too expensive to continue supporting
older versions of the REAPI protocol ?

[...]
> > If the APIs are not stable yet, I would propose that we consider a
> > bundled distribution of a version of BuildGrid that is known to work
> > with BuildStream 1.4, to be distributed *with* BuildStream 1.4, only as
> > a temporary measure until we can be sure that we don't break users in
> > the meantime.
>
> -1.  Using remote execution is optional.  It is implemented around a
> protocol.  There are multiple implementations of the protocol beyond
> BuildGrid: BuildBarn, BuildFarm, Scoot and Google's commercial Remote
> Build Execution service.  Each BuildStream user or group of users are
> free to chose which, and which stability guarantees they are
> comfortable with.

This interoperability is simply untrue until the specification
stabilizes, and our implementation of the REAPI as I understand it will
remain a step ahead of the specification for the foreseeable future.

Today for example, BuildStream will have to require workers which
clearly advertise their capabilities in terms of OS and machine
architecture of the execution environment, tomorrow it might be
something else, like full support for all file attributes on the remote
execution cloud.

I don't think it makes sense to deny users access to the feature just
because all implementations are not yet on board with the latest
version of the spec, or because we have not upstreamed all requirements
into the spec *yet*. We are still able to provide a reliable experience
for users who use BuildStream with our own implementation of the spec.


You are proposing that we start with experimental feature flags instead
which I think is a bit drastic, we have discussed experimental flags a
few times and have thus far remained adamant that it is not the right
approach for the project. As the approach leaves us open to multiple
pitfalls, such as:

  * Possible combinatorial explosion of enabled features which make
    the overall software potentially much more costly to maintain
    (potentially doubling complexity on a per experimental feature
    basis).

  * Situations where code is not road tested by real users, but mostly
    only tested by the developers of the experimental feature while
    being hidden behind an experimental feature flag.

In this instance it can be easily avoided by *temporarily* bundling
versions of the BuildGrid service which are known to work for a given
minor point BuildStream release line - i.e. users have the same
guarantees that they have with the artifact server, at a fairly low
cost.

In other words, lets not block the whole feature away from prime time
just because the REAPI spec is not ready, we really only need to block
interoperability guarantees until that time.

Cheers,
    -Tristan