[BuildStream] BuildStream Versioning & Releases, WAS: Re: bst CLI design and consistency (UI)

[Sander Striker <s striker striker nl>]

Hi,

Given most of what we're talking about has more to do with versioning, specific to BuildStream, I've changed the subject to reflect that.

Tristan, I know you care a lot about the project, please take this as a constructive perspective on how I see where we are.

On Sun, Dec 9, 2018 at 10:23 AM Tristan Van Berkom <tristan vanberkom codethink co uk> wrote:

Hi Sander,

On Fri, 2018-12-07 at 14:23 +0100, Sander Striker wrote:
> Hi,
[...]
> > While I do agree with Paul's adjacent reply here that breaking the
> > command line for us is much less of a concern than breaking format
> > compatibility, I think this is a temporary situation due to our tool
> > being relatively young
>
> It's young like pre-1.x :).

I believe this was a playful statement on your part, and I don't think
you mean to offend with it, but I honestly resent that people are
treating BuildStream as if it were some kind of unstable, unreliable
work in progress.

You're not going to like what I have to say I'm afraid.  My intention is to create more clarity on my perspective.  I am fairly certain that I'm not alone with this position.

Looking at other threads, bst pretty much is pre-1.x.  And we should be more frank and realistic about this.  I suggest reading on for why I say this.

We have a reliable format, we are feature complete for a core set of
features and we have users relying on the stability of this core
feature set.

I can't argue with use.  The definition of core set of features is a different story.  Use of semantic versioning schemes set expectations on everything though, not just a core feature set: https://semver.org/.  I'll also refer to http://apr.apache.org/versioning.html, which is worded less formally.

 

Making statements that we are "pre-1.x" only goes towards sewing
distrust in our existing user base, who have real needs, and to whom we
need to demonstrate our reliability.

Quite the opposite.  Making guarantees that we cannot and should not make is what creates distrust down the line.  Much like what is happening with 1.4 right now in which we chose to break things.  And that opened the door for taking a real critical look, and breaking more things to improve the experience.

>   Getting stable on the data format and cache keys early, would make
> life easier, as even with breakage in a CLI, versions would
> interoperate.  As such you could test new versions, decide to
> downgrade, and not lose the builds that you did with either version.

Stability of cache keys was *never* a guarantee, I'll re-post the link
to the original set of API stability guarantees:

    https://mail.gnome.org/archives/buildstream-list/2018-January/msg00006.html

Was that the correct link?  I'm not sure what I am supposed to take away from that.

If you did post a link to a message, I question what user is going read that do you think?  If as a user I find a 1.x release, that comes with certain expectations.  And if it doesn't, then I go back and read about what this specific project meant with their version  numbers, and likely be disappointed.

Frankly, changing cache keys does not make BuildStream backwards
incompatible, it only means that when you change a minor point version,
you will need to rebuild everything.

That's no minor thing though.  If I were an unsuspecting user, this would surprise me on 1.x releases.  And the first thing that would spring to mind would be that it had a bug.

It is a goal to reach this artifact stability, but it does not prevent
us from having a stable API.

If stability has exceptions, then calling it stable is a stretch at best.  It's then better to label as 0.x and be specific about what is stable, rather than the other way around.  Unfortunately we already burned the 1.x major number putting us on the path of burning more major version numbers.  Or declaring bankruptcy on 1.x.

There is a good reason that bazel is still at 0.x, being 4 years in.  It took subversion 2.5 years to go from self-hosting to a 1.0 release (http://subversion.apache.org/docs/release-notes/release-history.html).  Apache portable runtime took 2 years and change IIRC to go from 0.9.x to 1.0.

> > (I'm sure that we would all be pretty upset if
> > the git developers were to bikeshed their CLI even once let alone
>
> I think we can all agree that the world would have been a better
> place had that happened and introduced consistency.

I am honestly unfussed about my personal taste of how I might think the
git CLI *should* have been, which probably differs from whoever else
you ask.

I maintain that it is mostly a matter of taste, and the more offensive
thing would be for the CLI I have already learned and built programs
which incorporate it as a tool, to change it's behavior and interface.

Which is why mass adoption cements interfaces, and makes them unchangeable forever.

> > multiple times, given the amount of scripting which exists in the wild
> > and depends on the git CLI).
>
> You consider the current CLI work bikeshedding?

It doesn't really matter what I think because a vast majority of the
contributor base wants to overhaul the CLI.

My personal opinion is that it is definitely bikeshedding.

 

There will always be opinions about how the CLI could be better, and a
point comes when you need to put reliability and stability above
personal preferences and perfectionism - for BuildStream this time was
last year, and the fact that we are revisiting this worries me that it
might even happen again in the future.

The time wasn't last year.  The only argument for that is coinciding with when 1.x was created, which is circular.  There were no real user learnings yet.  BuildStream wasn't feature complete, and still isn't, so the impact of those unimplemented features was/is unknown.  

At the time I indicated that I didn't care so much about the releases.  That was a mistake, given the implications of a 1.x release.  I've recently repeated that mistake.  I've come to see that 1.x releases in this project slow us down to the point of grinding to a halt.  Changes are postponed because they have implications for the 1.x release.  Changes are discussed to the point of exhaustion because they change behavior.

I should have taken the position earlier that I care so much about releases that I think we shouldn't be doing them.  At least not beyond providing convenience snapshots for early adopters to use and try out.

I am however optimistic that the current set of contributors recognize
this, and that is why we are concentrating on it so vigorously, because
we recognize that we should never do it again.

I think this is much more fundamental.  I think the CLI design patterns will not change much beyond this.  However I still would consider bst to be pre-1.x quality or stability.  We score really poorly in terms of performance.  And I don't think we are there in terms of either cache key or artifact format stability.

That really is my own personal opinion on the matter, but as I said,
the majority of contributors really want this change and as such I am
not going to try and stand in the way of it.


That said, since the scope of these changes has gone so far, it is
really worth considering just calling the next release 2.0 (at first it
was just one or two workspace commands which were going to change).

I think that is a mistake and will put the 2.x series on the same path as the 1.x series.  This proposed 2.0 would still suffer from not being performant enough and from not having e.g. cache key stability.  Which, for what it's worth I'm only using as examples.

[...]
> > I'd like to also underline that our motivations for remaining stable in
> > the long term should be a matter of dignity, it is what makes the tool
> > trustworthy for the new users we want to attract - in other words: the
> > volume of existing users should not be the motivating factor in being
> > stable.
>
> I don't think anyone wants to break anything for the sake of breaking
> things.  I think we want to break things because the project is young
> and is still making mistakes/learning lessons that are worth
> correcting early.  And while yes, we should consider users, let's
> also be honest in that our current user base are of the early adopter
> kind.  They can take a bit more churn for the sake of progress.  As
> the project matures, our user base will change and our APIs, UIs, etc
> will effectively cement and become unchangeable.

I agree and disagree.

Look at it this way: We have a hand full of users which we have a close
relationship with; GNOME and freedesktop-sdk, as much as it does pain
them when CI breaks with an upgrade, our relations are good enough such
that we take care of their needs and overall we overcome the small
breakages.

Exactly.  And I posit that with convenience snapshot releases these audiences could have been similarly served.

On the other hand, over the last year we have had some random people
come into the channel, report an issue, and take the time to thank us
for our work, it really is a good feeling when people you dont know at
all are using the work.

Sure.  And we should signal to these people what the state of the project is, with full honesty so that they don't get the wrong impression.  And if we do they are either going to end up in the engaged-early-adopter category, or as a short lived user.

 

I think we should be even more proud about the successes we've had with
the users who show up out of nowhere, report issues and thank us, than
we should about the people we've worked closely with.

I think we should be proud when we have built a cohesive and supportive community that is eager to address the problems in this space.  That should result in a product to be proud of, and which users want to use because it is so good.  And because they have heard it is really good.  They will not come back just because the API is so stable between early releases; they will come back and get involved because the tool solves their problem well.  And once there are enough users coming back because we've reached this point, we should cut our stable release and give all the guarantees of a major version, without exception.

Even if these users are but a small hand full at this point, how our
breaking changes effects these people makes the largest impact in terms
of perception, and what these people are telling their colleagues goes
a long way to determining the future of our project.

Right now consider the possibility of these people telling their colleagues that a project that had two stable releases ... <insert short-coming on the dotted line>.

Cheers,
    -Tristan

Cheers,

Sander

--

Cheers,

Sander