Re: [BuildStream] Scheduler optimization

[Jürg Billeter <j bitron ch>]

Hi Benjamin,

On Thu, 2019-03-28 at 09:30 +0000, Benjamin Schubert via buildstream-list wrote:
> TLDR: I want to refactor the queues and scheduler from their current
> model to a push based model.
> I would need feeback on the approach to see if I'm not missing something.
> I would also need to know if there is any objections to my plan since
> this would be a significant refactoring effort.

Thanks for this write-up. The approach makes sense to me and I largely
agree with what Tristan wrote, so won't repeat those points.

While we're changing the scheduler architecture, I'm wondering whether
we should also consider implementing/supporting a make job server (or
at least make sure it would be possible to support this in the future).
This would likely improve scheduling quite a bit for local execution.
And the push-based model would make this a much better fit, i.e., it
matches the direction we're going.


> 4) When calling "set_required" on element, put the element in the fetch_queue.
>     - Currently, what happens is that the element is already in the queue, and when
>       "set_required" is called, a boolean is toggled, making the status of the element
>       go from WAITING to READY, and allows the queue to pick it.

"set_required" just means that the artifact is required. It doesn't
mean that the Element needs to be built and thus also fetching may not
be needed. I.e., "schedule_assemble" is probably closer to the right
trigger.

> 5) When an element is put in the build queue, if all its dependencies are not ready,
> add the element to a list in every dependency that is not ready. Whenever an
> element becomes done in this queue, go over this list and for every ready
> element put it in the queue.

Can't we use the recently added reverse dependency lists for this, at
least as a starting point? It might be slightly less efficient but we
anyway already notify all reverse dependencies to trigger cache key
calculations, so it might not make a big difference and could be
simpler, especially with regards to the separation between Element and
scheduling logic that Tristan mentioned.

Cheers,
Jürg