Re: [BuildStream] Proposal: A small number of subprocesses handling jobs

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Tue, 2019-03-05 at 16:55 +0900, Tristan Van Berkom via buildstream-
list wrote:
> Hi,
[...]
> Both of these drastic changes incur some measure of overhead.
>
> Where is the evidence that these overheads are going to be less than
> the overhead of simply forking on demand ?
>
> Without this evidence, we should definitely simply stay with option 3,
> which is to not make any changes, the code is simpler now than it would
> be if either of the above proposed routes were to be followed.

Actually, I should raise that a simple measurement is not sufficient in
this specific case, we would not be comparing oranges with oranges.

Even if the overhead of forking on demand turns out to be slightly
higher, the overhead of forking is linear; in the very worst case we
saturate a single process with the overhead of constant forking (at
scale), we *could* eventually separate the UI from the scheduler with
much more ease in order to allow greater parallelism if forking becomes
a bottleneck.

Additional synchronization of state across processes will not scale
cleanly when orchestrating hundreds or thousands of builds in parallel,
while forking on demand will inevitably hit a bottleneck, it's
performance will obviously scale linearly (and we avoid a potential can
of works when dealing with delicate issues around synchronizing data at
scale).

Cheers,
    -Tristan