Re: [BuildStream] Buildtrees in 1.4 release

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Laurence,

On Thu, 2018-11-29 at 21:47 +0000, Laurence Urhegyi via BuildStream-list wrote:
> Hi,
>
> I think it's time we discussed buildtrees and whether to include this 
> functionality in the 1.4 release. I have had multiple conversations in 
> private channels about this and think it's right to bring these onto the 
> list.

To me, this opening statement on a public list reads like someone
yelling "fire" in a crowded theater.

I don't think you realize this or that you intended it this way :)

> In my simple mind, we can summarise the problem like this:
>
> Buildtrees are generally always large and thus cause big problems when 
> cached due to their size. This is annoying for users. There are three 
> aspects of buildtrees to consider:
> * creating
> * pushing
> * pulling
>
> There have been discussions since the feature was originally conceived 
> over whether the above should be optional or happen by default. Having 
> downloads be optional has landed [0], but creation and pushing have not. 
> There's a ticket for making the uploading of buildtrees configurable [1] 
> (and some thought gone into it) but I don't think there is one for 
> creation.
>
> If we're being honest, it's looking unlikely that we'll be able to set 
> configuration options for creation and pushing in time for 1.4.
>
> Is this really acceptable to users, to not have these be optional?
>
> I'd like others to weigh in here but I feel like hiding this feature 
> with an experimental flag is the right thing to do for the  1.4 release.

A few things to say here:

  * Enabling build tree creation to unblock the surrounding feature
    work was the first thing we did when early branching for 1.2

    Which means we have had more than an entire release cycle to
    make this work perfectly - having to back this out now, would be
    very sad.

  * I am doubtful of the importance of any option for *creation* of
    build trees, especially now that we have local cache cleanup
    in a robust enough state.

    In other words, the local cache only needs to be large enough
    to store the sources, build trees and build results for a single
    full build - this is the same amount of local disk space required
    for a full build using say, JHBuild.

    Has anyone even raised an issue or proposed that *creation* of
    the build trees should be optional ?

  * If there are other features that we've been planning for 1.4
    much later in the cycle that are not landed yet, it makes a lot
    more sense to postpone those in order to ensure what we've already
    landed works seamlessly, than to start backing out previous
    features.

Finally, I wanted to make a statement soon about the release dates but
didn't judge that people were going to be panicking this week yet.

The point is that, we're trying to move the release cycle to be earlier
in order for our releases to be more conveniently *before* our regular
hackfests and meetups, rather than releasing directly after our
meetups.

This totally makes sense for the sake of the meetups, but the quality
and reliability of the release is certainly more important than the
convenience of our meetup.

Maybe we were overzealous in moving the release dates too early too
quickly, and it will make more sense to keep it at the regular date for
1.4 (I think that puts release day in early March and feature/API
freeze in February).

It makes no sense for people to be developing in a state of panic, and
since we did try to change the release dates in mid cycle, not doing so
this cycle is a lot less drastic than backing out features.

Cheers,
    -Tristan