Re: [BuildStream] Proposal: A small number of subprocesses handling jobs

[Jonathan Maw <jonathan maw codethink co uk>]

On 2019-03-05 08:39, Tristan Van Berkom wrote:
> Even if the overhead of forking on demand turns out to be slightly
> higher, the overhead of forking is linear; in the very worst case we
> saturate a single process with the overhead of constant forking (at
> scale), we *could* eventually separate the UI from the scheduler with
> much more ease in order to allow greater parallelism if forking becomes
> a bottleneck.
>
> Additional synchronization of state across processes will not scale
> cleanly when orchestrating hundreds or thousands of builds in parallel,
> while forking on demand will inevitably hit a bottleneck, it's
> performance will obviously scale linearly (and we avoid a potential can
> of works when dealing with delicate issues around synchronizing data at
> scale).
>
> Cheers,
>     -Tristan

Hi Tristan,

AIUI, a model where there is a single process acting as an element graph
service, while the workers use IPC to read from / write to the graph, 
would
not have the quadratic effort required to synchronise jobs.

Am I correct in this assumption, and that it would be approved if it can 
be
proven to be more efficient than the current model?

Best regards,

Jonathan

--
Jonathan Maw, Software Engineer, Codethink Ltd.
Codethink privacy policy: https://www.codethink.co.uk/privacy.html