Re: [BuildStream] Proposal: Decouple source tracking and building

[Chandan Singh <chandan chandansingh net>]

Hi Jürg, Tristan,

Thanks for the prompt replies. I'll reply to some of your comments inline.

> While I agree with the thought in general, I don't think the "having
> to type two commands" issue should just be dismissed. The reason James
> and I did not suggest this change was precisely that - by running it
> as two commands, we miss out on parallelism when tracking a large
> system.
>
> This might seem like it doesn't matter at first glance (because large
> projects will probably manually track source versions anyway), but I
> think it might be a problem for small projects that depend on large
> junctions, since the majority of their build time can be waiting for
> BuildStream to figure out whether all their junctions are up-to-date.
>
> In practice I think that might be rare enough of a situation that we
> can stop supporting it, but I wouldn't want to just dismiss the
> thought.

Tristan, I didn't mean to be dismissive of this use case. I was just
trying to figure out if I missed sometthing else.

You are definitely right that this would mean that we lose out on
parallelism (as Jürg also poined out). But, I am not sure if that is
realistically an issue. In the case you mentioned - a small project
depending on a large junction - I'm not sure how this change would
affect that. If you depend on that junction, BuildStream needs to
fetch it regarless of whether you are tracking anything or not.  Or
have I missed something?

Also, I'd expect the time to track elements to be relatively
insignificant compared to building them, be it small projects or
large. That may not be true in absolutely all cases. But, sas you say,
exceptions to this rule may be rare enough that we can stop supporting
them.

> > A possible middle ground could be a `bst-track-build` script in contrib
> > (or a track-build` command in bst itself) that would provide the
> > convenience of a single command for common cases without imposing
> > complexity in the main codebase. That's assuming the loss of
> > parallelism is not a significant concern.

Jürg, I am not stricly against this idea. But, I think we should only
do this if there is real demand for it, and not in anticipation. If
this script is going to literally run `bst source track` followed by
`bst build`, I'm not sure if that's really worth creating a wrapper
script.

Thanks!

Chandan