Hi all,
I've been meaning to make a statement over the past couple weeks, as
we've been seeing the pressure rising, but only finally got around to
writing it up today:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00039.html
Hopefully backing out of our rash decision of advancing the release
cycle in mid-cycle will relax the situation.
I should add, this is not only for the sake of build trees, we planned
to make a lot of breaking CLI changes which ideally we will never,
ever, ever do again;
Theoretically, yes. When I hear a desire to never release 2.0, I have very strong doubts.
What is considered to remain stable vs not is selective, as per the cache key stability thread. In other words just looking at the version number doesn't inform the user fully in terms of expectations.
and we should land these mostly simple changes
once and for all in 1.4; what is killing us in landing these features
is not the functionality itself but the tiresome bike shedding which
surrounds them, unfortunately.
Can you be specific in what features you consider bike shedding happening? And if you recognize this I am assuming you don't care about the color of the shed?
But back to the topic...
On Fri, 2018-12-07 at 00:58 +0100, Sander Striker via BuildStream-list wrote:
> Hi,
>
> On Tue, Dec 4, 2018 at 1:04 PM Tom Pollard via BuildStream-list <buildstream-list gnome org> wrote:
> > Hi all,
> >
> > On 30/11/2018 06:28, Tristan Van Berkom via BuildStream-list wrote:
> > <snip>
> > > A few things to say here:
> > >
> > > * Enabling build tree creation to unblock the surrounding feature
> > > work was the first thing we did when early branching for 1.2
> > >
> > > Which means we have had more than an entire release cycle to
> > > make this work perfectly - having to back this out now, would be
> > > very sad.
> >
> > After working on features around build trees I obviously share this
> > sentiment, regardless of my position of them being included in the release.
>
> As one of the original requesters of the feature, and having seen us
> wreck our brains on how it was supposed to work since January, I can
> relate. However, I don't think we should introduce behavior that
> would hurt users more than they would gain. And especially not when
> we are anticipating a behavioral change in the following release.
The creation of build trees is so deeply entrenched into the code that
I really don't think backing it out is an option, or rather, it is the
most expensive option.
I'm not convinced of that at all; it would surprise me if it took more than a day to make build trees disappear.
As everyone knows I am spiteful of any talk about "experimental flags",
lets use it at least as a strawman here: If we were even to consider an
experimental flag where everything was "optionally turned on", this is
even more technically challenging in my view than adding the missing
optionalities that are in mid flight (not only are downloads and
uploads, and possibly even creations optional, but the whole thing is
optional again - as well as dependent features which should "disappear"
when the experimental flag is turned off).
It is exactly the strict guarantees that you wish to give with versioning that result in the suggestion of an experimental flag. I don't want to be having the conversation with you for the 1.6 cycle that making build trees optional is going to be a change in behavior, and therefor cannot be done.
Realistically speaking, implementing any remaining optionality and
committing to it, is much less expensive than either backing out the
feature and its dependents, or making the whole thing optional.
You've now repeated this statement, consider my response a repeat as well. On top, the mumblings about it being hard to make creation optional, don't convince me at all.
> > > * I am doubtful of the importance of any option for *creation* of
> > > build trees, especially now that we have local cache cleanup
> > > in a robust enough state.
>
> I think you're forgetting about remote execution. If we
> unconditionally create the build trees it means we need to
> unconditionally capture these files on the worker, get them uploaded
> into CAS. And then we happily pull all of this stuff down into the
> client in the current implementation. And then we create the local
> artifact, which we then decide not to push?
> Clearly this is wasteful on resources, and not being able to turn
> that off seems just plain wrong. Especially if you know that you are
> not going to, or can not, use the build trees.
I am somewhat forgetting about remote execution, and while arguments
have been made so far that remote execution is added but should be
advertised as experimental for 1.4, I agree it would be nice to have
remote execution still "work well" in it's experimental state.
That said, the point I want to make here, is that making creation of
build trees optional is a commitment, but I think not making the
creation of build trees leaves the door open to making it optional
later.
Given our discussions on behavior changes I don't really want to postpone this. If the partial artifact discussion was easier, I wouldn't have this concern so much.
As such, I think that considering this early commitment as a blocker of
the feature, on the sole grounds that a new experimental feature might
perform better, is the wrong move.
I'm sorry Tristan, but it goes a bit further than this. Making build trees mandatory does make local builds more expensive as well. If someone would run the benchmarks I'm pretty sure that we can see that (unless IO suddenly became free).
That is not to say that the optionality of creating/caching build trees
is not otherwise beneficial, just that we can postpone committing to
this optionality if the only reason to commit to it is for a feature
that is still in the process of materializing.
I wish we cared a bit more about performance. And that in the case where users do not want to incur the cost of creating/storing/pushing/pulling, we shouldn't make them pay.
> > > In other words, the local cache only needs to be large enough
> > > to store the sources, build trees and build results for a single
> > > full build - this is the same amount of local disk space required
> > > for a full build using say, JHBuild.
>
>
> > > Has anyone even raised an issue or proposed that *creation* of
> > > the build trees should be optional ?
>
> My choice of words may not have been perfect, but I would
> consider https://gitlab.com/BuildStream/buildstream/issues/566 to be
> it. In that we also explicitly state that "Yes we are certainly on
> the same page here - I do want to see this used a lot before
> introducing an escape route, but agree that we should design one at
> some point before freezing things in 1.4." That was three months
> ago.
Yes, this was always discussed in terms of making the uploads optional,
not the creations. If there is something in the comments which speaks
of making creations optional as well as uploads, then I'm sorry that it
escaped me.
It is literally the first words in the issue: "The build tree should be optional in the artifact." And seriously, having to spell out "don't create a build tree in an artifact if you are not going to use it" seems excessive to get the point across. I think you can safely replace "upload" with "create" in
#566's summary.
[...]
> I don't disagree that we should refactor complexity. However, I
> don't think we should be making a trade between a bit of complexity
> vs changing the feature to now require build trees everywhere instead
> of having them be an optional for certain use cases.
The optional upload is already a work in progress, and we need to
decide on whether the creation is optional, and if that optionality is
tied into the same configuration as the optionality of uploads or not.
We have time to complete this optionality, and this optionality is less
expensive than an additional blanket optionality (experimental flag
strawman above) or backing it out entirely, I really think pushing
forward is the smallest effort of any of the choices.
Repeat three of this statement.
That said, all of this talk is secondary to the elephant in the room;
which is that very soon the build trees are *going to be mandatory*
dependencies for elements which run `make check` on the build trees the
necessarily depend on:
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00042.html
I still have my doubts that this is the right approach for testing; I may have only mentioned this in a private setting currently, mostly for opportunistic/time constraint reasons. Creating dependencies on build trees from another element I particularly frown upon. It's a dependency type we don't even support yet, and I question whether we should support it. However this goes into the subject of testing and veers away from what we are talking about here.
While postponing optionality is possible if the negative impact is not
too bad in local execution (no need to commit while experimental
features are still in flux), the conversation of how this commitment
impacts the future plans for the testing elements I think is
inescapable.
Well... unlike other features testing currently is a discussion, not an implemented feature. And nothing experimental here, just something non-existent. So while I agree we should not paint ourselves into a corner knowingly, I find it awkward to see this raised as a blocker.
We need a statement of how to reconcile:
* I have a project with elements which explicitly depend on build
trees in specific elements
* I want to turn off build trees completely
* I want to turn on build trees for specific elements and/or specific situations (e.g. a failed build).
Unfortunately, saying later that "turning off build trees" only means
"turning off the build trees which are not required" is very unwieldy
in practice; making the distinction of "a required build tree" is
almost impossible, it is like the having the past be informed by the
future.
I really have no idea what you are trying to say here. In the hypothetical case that I want to create an element with a dependency on another element its build tree, I will need to turn on the creation of build trees for that element. No time warps necessary.
Even if we loaded a whole project every time in order to determine in
advance if any *potential* elements explicitly depend on a build tree
and caused that build tree to become mandatory, it would be impossible
to determine that the test elements which you decided to maintain in a
separate BuildStream project, which depends on the build trees of the
source project across a junction, exists or not (i.e. because of
junctions, it is quite literally impossible to presume to have any
knowledge of reverse dependencies at any time).
Or you just error out with the message that a dependency couldn't be satisfied when building this element with this hypothetical dependency.
If we can answer the question of how to reconcile this plan of build
tree optionality, with the plan of explicitly depending on build trees
in test elements, then I'm sure we can solve this in time.
I consider this a blocker for any testing proposal. Build trees should never be mandatory, a design assuming this is just fundamentally flawed.
Cheers,
-Tristan
Cheers,
Sander
