[BuildStream] Thoughts around plugins (Too long, please read carefully)

[Daniel Silverstone <daniel silverstone codethink co uk>]

Hi all,

This is meant to be a starting point to think about various aspects of the
BuildStream plugin model.

Some of this is pie-in-the-sky, some is very odd, and some is more reasoned
thinking.  My goal here is to get all of this written down, and available to
discuss over time.

For the most part, I imagine this will happen in the 1.4 -> 1.6 development
period, but where I think things might be possible and/or desirable in the 1.4
timeline I will note it.  All of this is coming from my tortured mind and so I
apologise if it's a bit brain-dumpy, but I wanted to ensure we had time and a
seed to discuss from.

I am basically aiming at ensuring that, when plugins migrate out of core into
other repositories, or when plugins migrate among external repos, there exist
ways to communicate deprecation etc. to users, and to allow users to
communicate to us the exact requirements they have for particular plugins.

---

To start, my understanding of how plugins are acquired is as follows.  If any
of this is horrendously incorrect then please use the delta between this and
reality when interpreting my suggestions/ideas.

a. Plugins are acquired from three possible places:

1. The core (automatically available, versioned by means of the
   `format-version` in the `project.conf`.
2. Acquired from a path relative to the `project.conf` as `local` plugins.
   Versioned by means of the dictionaries in `project.conf::plugins`
3. Acquired from the system in some fashion, by means of `pkg_resources` (the
   `pip` source, but without auto-installation AFAICT).  Versioned by means of
   the dictionaries in `project.conf::plugins`

b. You may **NOT** have two plugins of the same `(kind, name)` used in the same
   project

c. When looking up a particular plugin name, first search the external plugin
   sources in order, and then try the core plugins.

d. While the documentation states "Note that plugins with the same name from
   different origins are not permitted." in reality this means that "plugins
   are resolved as per 'c' and so you can't find shadowed plugins".

e. I assume that junctioned in projects' elements resolve their plugins by
   name via the junctioned `project.conf` rather than the top level one.

---

Desired outcomes of all this work (some detailed further below):

* Docker related plugins (source, element, maybe tools?) moved to a docker
  related repository which has its own cadence independent of `bst` and
  `bst-external` but likely following along with the 1.6 cycle just for
  consistency.

* Python related plugins, i.e. `pip` source and element plugins among others,
  to be deprecated in core in the 1.6 cycle, migrated to an external
  repository, to be maintained in semi-sync until 1.6 released, and then be
  removed in the 1.7/1.8 cycle.

* Plugins, wherever they are from, able to issue deprecation warnings about
  their use, to be silenceable from `project.conf` 

* Each source of plugins to be permitted to be given a "handle" or "name" which
  can be used to disambiguate plugins of the same name.  The name "bst" being
  reserved for Buildstream.  In the case of projects junctioning in other
  projects, the project name can be further used to path to a plugin.

---

Deprecation warning structure:

In order that the deprecation warnings be useful and informative, while not
irritating users too much, it should be possible for:

a. the warning to be issued only once per plugin per `bst` operation.
b. the warning to be silenceable uniquely from `project.conf`
c. the warning should contain enough information to identify exactly which
   plugin is issuing the warning, and where it came from, and what element was
   using it when the deprecation warning was issued.  If we could somehow
   gather deprecation warnings up and issue them with all related elements
   that'd be even more useful.
d. the warning should contain information on where to find replacement plugins
   suitable for use, along with a link to documentation on performing the
   replacement
e. The deprecation warning can be upgraded to an error which carries the same
   information but cannot be silenced and will terminate `bst`.

Deprecations are important to allow projects (`bst` *and* external plugin
sources) to move forward, supporting their users according to their own rules
on long-term behaviour.  By providing links to how to undo the deprecation
warnings, and also allowing for the user to disable warnings in their project
if they're aware of the problem but unwilling (for now) to do the replacement,
we minimise the friction generated when deprecating/replacing a plugin.

The "uniquely silenceable" property will require a way to path to plugins
in order to silence them, falling into line with one of the desired outcomes
above.

The upgrade-to-error capability is the final piece in the puzzle.  This means
that we can have the following cycles:

* In v1.4, plugin is present and works
* In v1.6, plugin is deprecated, warns, still works
* In v1.8, plugin is present, errors if used, doesn't work.
* In v1.10, plugin is gone, any attempt to use is simply "unknown plugin"

---

Pathing to plugins:

1. Every project has a unique name within an invocation of `bst`
2. Every source of plugins *may* be given a name in the `project.conf` of a
   particular project
3. As a result, we can extend the plugin name concept to be a path with some
   levels of optional prefixes.

I would propose that, rather than simply `$name` for plugins, we have something
which might be described as: `[$project_name//][$source_name/]$name` allowing
for the selection of specific plugins no matter how they come into the
awareness of `bst`.

Since the `core` plugins are automatically available in any project, the use of
a fully qualified `$project_name//bst/` prefix is redundant for those, though
not invalid.

Let's set up an example in order to think about how names are resolved.
Consider the following:

1. A project `app` refers to a plugin source called `stuff` which contains,
   among other things, a source plugin `appsrc` and an element plugin `elemoo`
2. The `app` project junctions in a `libs` project.
3. The `libs` project refers to a plugin source called `stuff` which contains,
   among other things, a source plugin `libsrc` and an element plugin `elemoo`
4. For the sake of this example, the two `elemoo` plugins are different in
   their behaviour to some extent, and cannot be harmonised.
5. The `app` project has, within it, an incubating library element which will
   eventually be offered to the `libs` project.

Many elements in the `app` project are able to use the `appsrc` source plugin
and the `elemoo` element plugin from the app's sources, simply by referring to
them by their given name.  `kind: appsrc` for example.

The elements in the `libs` project are able to use their `libsrc` and `elemoo`
plugins again purely by their given name `kind: elemoo` for example.

This is, so far, exactly what we currently have.

Now consider an element within the `app` project which is this library element
incubating there before being offered up to the `libs` project.  This would be
able to use the junctioned in plugin source along the lines of `kind:
libs//libsrc` or even `kind: libs//elemoo` (even though `app` has its own
`elemoo` element)

To extend this further, assume that the `libs` project has a second plugin
source they are experimenting with which they have called `stuff2` and which
contains an alternative `elemoo` element plugin which they are wanting to try
out to see if they want to migrate.  An element in `libs` could refer to that
as `kind: stuff2/elemoo` and an element in `app` which wants to try it could
use `kind: libs//stuff2/elemoo`.

In order to make this work, some small behavioural changes in `Project` and
`PluginContext` for the new `name` property for plugin sources would be needed,
and commensurate changes in `Project.create_{element,source}()` and
`PluginContext.lookup()` for the project-name prefix and source-name prefix
handling respectively.

---

Timeline for this work:

1. We need deprecation handling, in full, in place first.  In theory this
   could be achieved in the 1.4 timeline if we can agree on the format
   for doing so.
3. If we add the proposed plugin name resolution approach above next, there
   will be a principle of least surprise (current stuff works unchanged) along
   with the ability to bring in conflicting (in name) variants of plugins to be
   used in a single project.  While this *could* increase cognitive load when
   reading a project, if a project has named plugin sources and *always* uses
   the fully qualified name of the plugin, it should be entirely mitigated.
   Again, if agreed, I can see this being doable for 1.4, meaning that the
   rest of this timeline would work.
2. Once 1.4 is released, we could then migrate plugins out of the places they
   currently live into new locations, marking the old locations deprecated,
   with a pointer to how to acquire, and documentation on how to update to, the
   new plugins.  We can't do this before an official release which adds the
   deprecation mechanisms to the public API, and the name resolution proposal
   will improve matters for documenting how to migrate over time.
4. Before 1.6 is released, the python related plugins (and potentially others)
   are copied out of core into their new homes.  The copies in core are
   annotated with appropriate deprecation warnings once that has happened.
5. We keep any changes to the python plugins "manually" replicated between the
   two repositories for the 1.6 cycle
6. Once 1.6 is released, the core versions of the plugins are trimmed down to
   their pure skeletal form, along with changing the deprecation from warning
   to error.
7. Once 1.8 is released, the core versions of the plugins are removed entirely.

Along with the code changes as listed above, we will also need to ensure there
are suitable documentation pages available for plugin transitions, along with
documentation on how deprecation and removal is handled in a policy sense.  We
could then, as a core project, encourage the adoption of a similar policy by
any plugin sources which are not controlled by the main project.

Deprecation warnings, upgrades-to-error, and removals, should be
front-and-centre in the NEWS, on the release notes when a release is made on
the website, etc.

---

If you reached this far, thank you for reading all this, and I look forward
to discussing options, ideas, etc, with you.

D.

-- 
Daniel Silverstone                          https://www.codethink.co.uk/
Solutions Architect               GPG 4096/R Key Id: 3CCE BABE 206C 3B69