Re: [BuildStream] Scheduler optimization

[Jürg Billeter <j bitron ch>]

On Fri, 2019-03-29 at 16:45 +0900, Tristan Van Berkom wrote:
> On Fri, 2019-03-29 at 08:30 +0100, Jürg Billeter wrote:
> > Hi Benjamin,
> [...]
> > While we're changing the scheduler architecture, I'm wondering whether
> > we should also consider implementing/supporting a make job server (or
> > at least make sure it would be possible to support this in the future).
> > This would likely improve scheduling quite a bit for local execution.
> > And the push-based model would make this a much better fit, i.e., it
> > matches the direction we're going.
>
> Wouldn't it also make sense to consider job servers in the remote
> execution scenario ?
>
> I expect that most remote execution setups involve large servers where
> many remote workers are running.

Theoretically, yes. However, the BuildStream scheduler isn't running on
the remote workers, so this would be mostly independent. And it could
raise concerns of scheduling fairness across jobs from different
clients.

> (That said, I think we're a bit off topic here and think we should be
> careful to not lump too many changes into the same activity, it just
> makes sense to consider how jobs servers fit into a future where this
> scheduler change happens).

Yes, I think it's worth considering when designing the new scheduler
architecture, however, actual job server support would most likely be a
separate (and optional) effort after the core changes.

Cheers,
Jürg