Re: [BuildStream] Proposal: A small number of subprocesses handling jobs

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Tue, 2019-03-05 at 14:29 +0000, Jonathan Maw wrote:
[...]
> Hi Tristan,
>
> AIUI, a model where there is a single process acting as an element graph
> service, while the workers use IPC to read from / write to the graph, 
> would not have the quadratic effort required to synchronise jobs.

The service can only respond to one client at a time, and will
certainly incur a lot of context switches, causing many parallel
processes to block and wait for responses; with many processes making
requests at different timings in parallel I do suspect this to cost
more than a one-off payment of a fork()ed process.

Even if we set the above aside, the amount of state which needs to be
transferred over the wire will inevitably increase with dependency
*depth*: The larger the project, the more information about, e.g.
dependency cache keys, needs to be sent over the wire for each
individual process.

> Am I correct in this assumption, and that it would be approved if it can 
> be proven to be more efficient than the current model?

I have serious doubts that this can turn out to be true, and I
certainly think we should adhere to the age old saying; if it aint
broke then don't fix it, especially considering that most of the ideas
discussed here make things more complicated than they already are,
without any guaranteed benefits.

A more direct answer to your question would be: I think we should
seriously consider massive changes like this, even if they add a ton of
undesirable complexity, when the benefits can be proven, yes (and after
such prototyping and discovery would probably be a good time to start
discussion on how to sanely implement it).

Conversely, when it comes to proving huge changes like this will
actually pay off, and it takes a lot of effort to prototype and test;
we should not be expectant that the huge change will land just because
we spent a month or two trying to prototype and discover if it will
work: Trying these approaches in side branches is still a valuable
exercise, even if some of these approaches don't work and we never land
the code.

Finally: I think it would be beyond reckless to design this based on a
theory that it might be better and then proceed to land patches towards
this directly in master incrementally (this should go without saying,
but as last cycle we did a lot of "landing of unfinished stuff" in the
interest of unblocking parallel work, I think this needs more emphasis,
a core change like this is not the same as a high level feature
addition like build tree caching).

Cheers,
    -Tristan