Re: [BuildStream] Responsive, but not overly verbose UX - not-in-scheduler

[Jonathan Maw <jonathan maw codethink co uk>]

On 2019-06-07 18:04, Jonathan Maw via buildstream-list wrote:
> So, my understanding of how this would work for Queue
> active/processed/skipped/failed jobs is:
>
> Based on the principles of:
> * Stream *is* core, and provides a suitable interface that frontends
> can subscribe to
>   i.e. in a same-process model they can register callbacks to be
> notified when the state changes,
>   in a multiple-process model they would be given a python
> multiprocessing Queue, and would consume messages passed through it,
> though the details of what a multiple-process model would do don't
> affect my work at the moment.
> * Stream should hold all information that frontends need to access
> (i.e. isn't just a translation layer that accesses other parts of core
> when it needs to provide information)
>
> The changes to fit this model are:
> * A new `State` class (owned by Stream) would hold all the information
> we want to make available to frontends.
> * State holds a list of all active jobs (i.e. the job_start_callbacks
> and job_complete_callbacks that the Scheduler call are now defined by
> State)
> * State holds tallies of `TaskGroup`s, which hold tallies of how many
> jobs have been skipped/processed/failed.
>   - Since we only ever check the length of the skipped/processed
> lists, they may as well be integer counts stored only in State.
> * It is possible to register job_start_callbacks and
> job_complete_callbacks with State, so that frontends can have a list
> of all active jobs to print.
> * It is possible to register queue_status_changed_callbacks with
> State, so that frontends can report queue statistics.
> * It is possible to register task_groups_changed_callbacks, which
> report the names of all the queues now used by the Scheduler.
> * Since our frontend doesn't just print a notification every time
> something happens, it needs its own local representation of this data.

Hi,

Having made some progress in implementing this, I think I've reached a 
sticking point.
As-implemented, the methods in `State` to set parts of state are in 
exactly the same form as the callbacks I'd have frontends able to 
register in State, i.e. when `State.set_task_groups()` is called, it 
sets the list of task groups, and then executes a callback that would 
pass the list of task groups to the frontend.
This makes actually holding any state in State redundant, and I'm not 
sure if doing so serves any purpose.

This means my options are:
* Assume it'll make sense later (dubious)
* See if I can simplify it on Core's end by replacing callbacks that set 
State with direct accessors
* Not bother storing data in State, treat it as an Interface.
* Give the frontend direct access to State (possibly with appropriate 
getters.
  - in a multi-process model, this could be shared memory, with locking 
to ensure it doesn't get out of sync.
* Since State is a much simpler data model, periodically serialise the 
whole thing and send it to the frontend (probably not, seems very 
inefficient)

Since this is a pretty vague description, the current form of my 
implementation is at 
https://gitlab.com/BuildStream/buildstream/blob/33a09c5e91ebd086f9adf5c8d15163245cef4683/src/buildstream/_state.py

What do you think?

Thanks,

Jonathan

--
Jonathan Maw, Software Engineer, Codethink Ltd.
Codethink privacy policy: https://www.codethink.co.uk/privacy.html