Hi all,
On 30/11/2018 06:28, Tristan Van Berkom via BuildStream-list wrote:
<snip>
> A few things to say here:
>
> * Enabling build tree creation to unblock the surrounding feature
> work was the first thing we did when early branching for 1.2
>
> Which means we have had more than an entire release cycle to
> make this work perfectly - having to back this out now, would be
> very sad.
After working on features around build trees I obviously share this
sentiment, regardless of my position of them being included in the release.
As one of the original requesters of the feature, and having seen us wreck our brains on how it was supposed to work since January, I can relate. However, I don't think we should introduce behavior that would hurt users more than they would gain. And especially not when we are anticipating a behavioral change in the following release.
> * I am doubtful of the importance of any option for *creation* of
> build trees, especially now that we have local cache cleanup
> in a robust enough state.
I think you're forgetting about remote execution. If we unconditionally create the build trees it means we need to unconditionally capture these files on the worker, get them uploaded into CAS. And then we happily pull all of this stuff down into the client in the current implementation. And then we create the local artifact, which we then decide not to push?
Clearly this is wasteful on resources, and not being able to turn that off seems just plain wrong. Especially if you know that you are not going to, or can not, use the build trees.
> In other words, the local cache only needs to be large enough
> to store the sources, build trees and build results for a single
> full build - this is the same amount of local disk space required
> for a full build using say, JHBuild.
> Has anyone even raised an issue or proposed that *creation* of
> the build trees should be optional ?
My choice of words may not have been perfect, but I would consider
https://gitlab.com/BuildStream/buildstream/issues/566 to be it. In that we also explicitly state that "Yes we are certainly on the same page here - I do want to see this used a lot before introducing an escape route, but agree that we should design one at some point before freezing things in 1.4." That was three months ago.
I think it would be wrong to not include them (where applicable) in the
creation of artifacts, as I don't like the idea of supporting two
definitions of an artifact be it with an experimental flag or otherwise.
There would only be one definition of an artifact. The definition would contain an optional build tree. It would be compatible with older artifacts, which do not have a build tree.
However I think there's some merit in discussing if that means we
necessarily have to cache the buildtree locally after the creation and
the calculations that go along with it, in the same way we now allow for
pulling a buildtree-less artifact.
Indeed. The performance angle should not be ignored. The best optimization is not doing something.
The remote execution flow is a good one to consider when thinking about these issues, as per above.
As mentioned in other tickets/issues related to various buildtree
operations, I think a rework of element._update_state() is key moving
forward. Being able to track the 'partial' state of an element, as we
currently can in terms of cached & key types, locally without the need
to query the CAS in various areas that will only increase is something
that will make the divergence from 'the element is cached we have
everything' presumption more manageable.
I don't disagree that we should refactor complexity. However, I don't think we should be making a trade between a bit of complexity vs changing the feature to now require build trees everywhere instead of having them be an optional for certain use cases.
> * If there are other features that we've been planning for 1.4
> much later in the cycle that are not landed yet, it makes a lot
> more sense to postpone those in order to ensure what we've already
> landed works seamlessly, than to start backing out previous
> features.
+1. I think everyone can agree that this option would be the best, sadly
I'm not positioned to say if it is feasible. Making the uploading of
buildtrees (issue 566) is making progress in terms of the current state
of play and how it can be handled going forward, if this is can land
before the release are buildtrees in an acceptable place? We have
optional pulling bst-wide, the optional attempt to fetch missing
buildtrees into none pull/build commands that are configured to require
them, I feel like optional push being added to this makes quite good
coverage.
Regards,
Tom
Cheers,
Sander
> Cheers,
> -Tristan
>
> _______________________________________________
> BuildStream-list mailing list
> BuildStream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list
>
--
https://www.codethink.co.uk/privacy.html
_______________________________________________
BuildStream-list mailing list
BuildStream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list
