Re: [BuildStream] Responsive, but not overly verbose UX - not-in-scheduler

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Fri, 2019-06-07 at 18:04 +0100, Jonathan Maw wrote:
> Hi Tristan,
>
> (comments inline)
[...]
> So, my understanding of how this would work for Queue 
> active/processed/skipped/failed jobs is:
>
> Based on the principles of:
> * Stream *is* core, and provides a suitable interface that frontends can 
> subscribe to
>    i.e. in a same-process model they can register callbacks to be 
> notified when the state changes,
>    in a multiple-process model they would be given a python 
> multiprocessing Queue, and would consume messages passed through it, 
> though the details of what a multiple-process model would do don't 
> affect my work at the moment.
> * Stream should hold all information that frontends need to access (i.e. 
> isn't just a translation layer that accesses other parts of core when it 
> needs to provide information)

Umm I think you misunderstood something here, before I go on with the
rest of this message I want to check that it is not based on a false
assumption.

Let me clarify:

   The frontend and it's interaction with Stream() shell *never* be
   changed by a process separation of the internal scheduling process.

   All of this process separation shall be made completely transparent
   to the code in `_frontend/`, which only talks to Stream().

   Stream() decides how to carry out it's work, if Stream() decides to
   run the scheduler in a subprocess, that is it's business, but the
   code in `_frontend/` shall continue to just treat Stream as it would
   any other library, and receive callbacks in it's own process.

Does this change anything for you ?

The point here is that once you have made this separation so that
Stream() provides a clean separate interface for the frontend to drive
it's UI without needing to poke at core internals - the work that
others are doing for process separation will not concern the frontend
*at all*, it will just magically work.

I would expect that you would finish much earlier, and you would leave
the code in a state where the frontend has no ties to the internals
anymore. This way all of the work for process separation need no longer
concern itself with how it impacts the code in `_frontend/`, they must
only continue to honor the API contract set out by Stream() and ensure
the correct notifications are delivered in the main process.

  Exception: There is the detail of element state which still needs to
             be sorted out before clean separation is complete, but
             I think you get the gist.

There is an important thing to also consider which might be confusing:

  * In the discussions about separating the scheduling process, we've
    been referring to the main process as "the frontend process", this
    is the process which essentially the CLI runs and the code in
    `_frontend/` interacts with Stream().

  * In our conversation here, we've been referring to the frontend
    as "the code in _frontend/", as in "not the core"

Both definitions of "frontend" here are rather valid, but notice there
is a distinction here.

Cheers,
    -Tristan