Re: [BuildStream] Proposal: Decouple source tracking and building

[Jürg Billeter <j bitron ch>]

On Fri, 2019-10-18 at 14:55 +0100, Chandan Singh wrote:
> Hi all,
>
> tl;dr: Remove `--track-*` options from `bst build` and simplify
> codebase.
>
> ---
>
> [...]
>
> Decoupling tracking and build operations should simplify the
> `update_state()` logic as it would eliminate one of the needs for
> doing so.

Yes, I would definitely like this simplification. With a strict build
plan this could allow very simple state handling.

Unfortunately, non-strict builds (with remote artifact cache) would
still require support for strong cache keys not being available early
on. However, if we decouple tracking and building, it may be possible
to keep the complexity for non-strict builds in one place. An
alternative could be to attempt to determine strong cache keys early on
even for non-strict builds by fetching all artifact protos before
running the build pipeline (that would partially bring back #179 for
non-strict builds, though).

> [...]
>
> I can understand the desire to update all source references in a CI
> build, or for testing purposes. But, I think the same can be achieved
> by running`bst source track` followed by `bst build`. Can people
> think of any other use cases that would be negatively affected by
> this change, other than having to type two commands instead of one?

Besides the small loss of convenience, I can think of the following
disadvantages:

 * Loss of parallelism: Tracking and building separately obviously
   makes it impossible for BuildStream to already start fetching and
   building resolved elements while other elements are still being
   tracked. I assume this is mainly a concern for larger projects when
   tracking many or all elements. However, compared to the overall
   build time of a large project, the time required for tracking is
   likely still relatively small, so it might not be a significant
   concern.
 * Delayed error messages: E.g., if I enter `bst source track foo.bst
   && bst build goo.bst` (typo) in a shell, see that BuildStream is
   starting to track, and then walk away, I won't get an error message
   until tracking has completed. Could be an issue with more complex
   build command lines. However, if you repeatedly run complex
   combinations, you probably don't want to manually enter it all the
   time anyway. Also not a real issue for the CI use case.

For my own builds I typically prefer separate tracking as I want to
review the changes applied by `bst source track` before starting the
build (except for the CI use case).

Overall I'm a big proponent of reducing complexity and unnecessary
interdependencies where we can, so I'm leaning towards being in favor
of this proposal. However, UX convenience is also important, so it
depends on whether there are common use cases where this would really
be a step backward.

A possible middle ground could be a `bst-track-build` script in contrib
(or a track-build` command in bst itself) that would provide the
convenience of a single command for common cases without imposing
complexity in the main codebase. That's assuming the loss of
parallelism is not a significant concern.

Cheers,
Jürg