[BuildStream] Proposal: Decouple source tracking and building

[Chandan Singh <chandan chandansingh net>]

Hi all,

tl;dr: Remove `--track-*` options from `bst build` and simplify codebase.

---

I wanted to propose a clearer separation between tracking sources and building
them in BuildStream. Currently `bst build` offers a few `--track-*` options,
all related to whether or not BuildStream should attempt to track sources
before starting the build.

In my opinion, these are two different kinds of operations, and don't really
need to be part of the same pipeline. My concerns about this can be broken down
into two broad areas:

1. Simplifying scheduling/update_state() logic
2. UI/UX

First, let's talk about scheduler. At present, when we start a build, we expect
the elements being tracked to be resolved only after they've been tracked. This
means that we can't be sure that all elements in the pipeline are fully resolved
and ready to be built.

Decoupling tracking and build operations should simplify the `update_state()`
logic as it would eliminate one of the needs for doing so.

Moving on to UI/UX issues, `bst build --track-*` essentially duplicates
functionality from `bst source track`. Hence we end up with two ways of doing
the same thing and have to support both.

Since tracking sources and building are rather different kinds of operations,
I am proposing to have them clearly separated. More concretely, I am proposing
to remove the following options from `bst build`:

  * `--track`
  * `--track-all`
  * `--track-except`
  * `--track-cross-junctions`

  * There's also `--track-save` which is deprecated and ignored

`bst source track` already supports all functionality offered by the above
options, so that won't require any additional features.

I can understand the desire to update all source references in a CI build, or
for testing purposes. But, I think the same can be achieved by running
`bst source track` followed by `bst build`. Can people think of any other use
cases that would be negatively affected by this change, other than having to
type two commands instead of one?

Let me know what you think.

Thanks!
Chandan