Re: [BuildStream] Proposal: A small number of subprocesses handling jobs

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Mon, 2019-03-04 at 18:02 +0900, Tristan Van Berkom via buildstream-list wrote:
> Hi,

[...]
> I would personally rather be reluctant about imposing this explicit
> parallelism knowledge on the plugin API, and seek other
> justifications (performance ?) before making plugins aware of what is
> processed where.

So here is another idea...

What if we elaborated and generalized further on state synchronization,
such that:

* We had a dedicated process pool
* Plugins are run in the process pool
* The Element/Source API used in a worker pool would obtain state from
  the main process on demand, such that we load state lazily on demand
  by querying the main process.
* The core's responsibility then is to:
  * Dispatch jobs in worker threads
  * Respond to generic get/set requests to the data model in the
    main process
  * Update the UI periodically

This approach might incur a lot more context switching between
processes where plugins load state from the main process, but python
processing remains parallelized and the whole implementation is
transparent to the plugin.

Cheers,
    -Tristan