Re: [BuildStream] [Summary 2] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi all,

So here is another summary after the tail end of the conversation, as
requested by Sander.

The following represents a series of steps regarding how we proceed
with migration of plugins:

  1) Move all plugins that are not hardwired into the core into
     a single separate repository.

     NOTE: I think at this point it is worthwhile to double check that
           we are satisfied with our `testing` API, and that we dont
           expose too much API here as it will become a stable API 
           surface in BuildStream 2.

           Any changes we make to the testing API while it remains
           unstable will cause more churn if we have many different
           plugin repositories.

  2) Move plugins into separate domain specific repositories.

     Consequences of fragmenting plugins will be that:

     - We will not recommend distro level packaging of these fragmented
       plugin repositories.

       Even if we did, it would be a badly regressed user story, as
       the ramp up time for working on any BuildStream project on your
       own computer will require knowing what packages to install for
       that particular project (while currently most practical plugins
       a project needs come with BuildStream core, and a project might
       just introduce a couple of project-bound plugins).

     - We will recommend that project authors use `git submodules` (or
       other VCS equivalents) for the loading of plugins at the project
       level, and using the `local` plugin origin for loading these.

       This will effectively counteract the regression that users would
       need to install various moving parts in order to work on a
       BuildStream project on their machine.

  3) Examine at this point whether we need to evolve plugin loading.

     My opinion is really that we must, I don't think that the
     `git submodule` approach of loading plugins is catching on.

     If people decide to load plugins through the `pip` origin instead,
     that is rather fine for our upstream set of core plugins which are
     going to be strictly API stable (however I would still consider
     that a regressed user story, as the end user needs to know what
     to install).

     However, I am much more concerned about providing a reliable
     and safe experience for the *unstable* plugins we don't maintain.
     
     The whole point of plugins is to allow projects to do have the
     freedom to construct their artifacts in any way that they like,
     using the YAML for configuration and Sandbox for execution.

     If people work around the VCS approach for loading any project
     specific unstable plugins (possibly just because they find
     git submodules to be inconvenient, confusing or unwieldy, which
     many people do), then they will hurt themselves on junction
     upgrades when there really should not be any friction there.

     I think it's fine to allow people access to the sharp edges to
     cut themselves on, but I think we also have a responsibility to
     make the safest mode of operation also be the most convenient and
     obvious mode of operation.

I don't object with proceeding in this direction but still think it
will be important to automate the process of obtaining plugins.

Cheers,
    -Tristan


PS:

One of my objectives is to allow users to safely use snapshots of
BuildStream 2 along side their existing BuildStream 1 installations as
soon as possible, we need all the help we can get identifying issues in
the development of BuildStream 2 and ensuring we don't regress any use
cases.

It would also be nice to setup a public BuildGrid for people testing
out BuildStream 2 to help address issues there too.

I am a bit worried though about starting to make snapshots of
BuildStream 2 while our plugin story is in flux - it is alright to tell
users to "try this unstable thing" and it will churn a bit, but this
level of churn is a bit much even for people who buy into testing the
unstable thing.

As such, I hope that we can finalize the plugin story as soon as
possible and reduce ongoing plugin location churn.