Re: [BuildStream] Thoughts around plugins: deprecation

[Daniel Silverstone <daniel silverstone codethink co uk>]

On Thu, Nov 29, 2018 at 13:41:58 -0500, Chandan Singh wrote:
> Thanks for the detailed write-up. Please correct me if I am wrong, but
> it seems to me like the main motivation for your message was to figure
> out a strategy for deprecating  plugins, both core and external.

Yes, that's the majority of the drive of the posting.

> Most of the replies seem to have gone on a tangent about how to
> distribute and acquire plugins. While that is also an important topic,
> I feel like that is orthogonal to the deprecation discussion as no
> matter how users acquire plugins, there would still be the need to
> deprecate or move plugins from time to time. So, I would propose
> narrowing the scope of this thread to plugin deprecation and circle
> back to distribution separately.

Works for me :-)

> To that end, please find my replies inline for the deprecation proposal.

Thanks.

> > * Docker related plugins (source, element, maybe tools?) moved to a docker
> >   related repository which has its own cadence independent of `bst` and
> >   `bst-external` but likely following along with the 1.6 cycle just for
> >   consistency.
>
> While it may be true for the docker plugin that the deprecation will
> follow along the 1.6 cycle, I would like to emphasize that this may
> not always be the case. As the number of external plugin repositories
> grow, they might have their own release cadences.  So, if a plugins
> moves from one repo to another, it may not always coincide with a
> BuildStream release. I don't think this would necessarily affect the
> rest of the proposal in a major way, but I would like us to keep this
> in mind.

I figure that, for the most part, people providing external plugins will likely
only perform deprecation/functionality-shifts in line with bst releases, though
as you note, that's not necessary nor expected for everyone.

> > * Plugins, wherever they are from, able to issue deprecation warnings about
> >   their use, to be silenceable from `project.conf`
>
> +1

I'm glad you agree with this.

> > * Each source of plugins to be permitted to be given a "handle" or "name" which
> >   can be used to disambiguate plugins of the same name.  The name "bst" being
> >   reserved for Buildstream.  In the case of projects junctioning in other
> >   projects, the project name can be further used to path to a plugin.
>
> It was not entirely clear to me what is the motivation for this. So,
> perhaps you could clarify that. I have made an assumption about the
> motivation and replied below based on that.
>
> I suppose these names would be useful if half the elements in a
> project are using the deprecated plugin, while the others are using
> the new one. If that's the case, I am not sure if we really need names
> at this stage. When moving plugins around, I would hope that the first
> version in the new repo is identical to the old one so the projects
> can separate the process of switching to the new repo/package and
> upgrading to a new version.

I was thinking that, in order to be able to silence a deprecation warning we
need to be able to address the specific plugin it comes from.  Since during the
migration from one plugin to another (across repos) there may be a time when
more than one plugin has the same *basename* it's necessary to be able to
address them within their particular sources.

[snip]
> The deprecation warning structure seems sound to me.

I'm glad.  It was a point I was slightly nervous about.

> > The upgrade-to-error capability is the final piece in the puzzle.  This means
> > that we can have the following cycles:
> >
> > * In v1.4, plugin is present and works
> > * In v1.6, plugin is deprecated, warns, still works
> > * In v1.8, plugin is present, errors if used, doesn't work.
> > * In v1.10, plugin is gone, any attempt to use is simply "unknown plugin"
>
> I wonder if we can skip the "plugin is present, errors if used,
> doesn't work" state and directly jump from emitting warnings to
> removing the plugin. In some of the other projects I have worked with,
> it has been common to produce warnings to the effect of "this plugin
> is deprecated in this version and will be removed in the next
> major/minor release (x.y)". I am not sure if that step is adding much
> value, but does add administrative and maintenance burden on us.

We could skip the step, but my feeling is that not everyone moves through every
intermediate version and as such the longer we can provide *useful* migration
notes to users the better.

> > Deprecation warnings, upgrades-to-error, and removals, should be
> > front-and-centre in the NEWS, on the release notes when a release is made on
> > the website, etc.
>
> While this definitely makes sense for the core plugins, I am not sure
> how would this scale with external plugins for two reasons:
>
> 1. As mentioned above, the release schedules for external plugins
> might not always coincide with BuildStream release cycles.
> 2. It might be logistically difficult to collect all the warnings from
> external plugins as their numbers grow.

Oh I meant it for each relevant project in turn.  So moving something from
core to an external repo would warrant NEWS in the `bst` repo, but moving
something from `bst-external` to some other repo would warrant NEWS in
the `bst-external` repo instead.

A topic for the splitter 'distribution of plugins' discussion could be
aggregation of NEWS.

> > If you reached this far, thank you for reading all this, and I look forward
> > to discussing options, ideas, etc, with you.
>
> Thanks again for starting this conversation :)

You're welcome and thank you for taking the time to respond.

D.

-- 
Daniel Silverstone                          https://www.codethink.co.uk/
Solutions Architect               GPG 4096/R Key Id: 3CCE BABE 206C 3B69