Re: [BuildStream] Responsive, but not overly verbose UX - not-in-scheduler

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Jonathan,

On Tue, 2019-05-28 at 18:23 +0100, Jonathan Maw via buildstream-list wrote:
> Hi,
>
> I've picked up this task and am looking into ways to solve this problem 
> > (created https://gitlab.com/BuildStream/buildstream/issues/1029, which 
> has a summary of my understanding of the problem).
>
> I have also created a Merge Request with a very basic implementation of 
> progress reporting as I envision it at the moment 
> > (https://gitlab.com/BuildStream/buildstream/merge_requests/1358)
>
> I think the implementation will probably diverge from the proposed 
> implementation described in 
> https://mail.gnome.org/archives/buildstream-list/2019-April/msg00054.html, 
> specifically in regards to how it's presented to the user, as there is 
> currenlty ongoing work to move the UI into a separate process 
> > (https://gitlab.com/BuildStream/buildstream/issues/1036), so my primary 
> concern at the moment is to create a consistent and useful interface, 
> with a rough implementation in the existing UI that can be updated to 
> use the UI in a separate process later.

I think this is exactly the wrong thing to be doing.

As you pointed out, there is a huge amount of plumbing going in in
#1036 which we've spent the last week discussing.

This logging/responsiveness is basically polish, and doing it without
considering the incoming architectural changes is just going to add
friction to those who are implementing the process separation.

Instead of implementing more abstraction/clean separation between the
scheduler loader and the frontend, we would be hardwiring more
callbacks to run directly through the frontend; this would be worsened
if it is done without using the regular standard logging paths but
introducing new ones, all in all it's pushing in the opposite
direction.

> So far, my task breakdown is:
> * Report progress when loading elements (read file to make a 
> MetaElement)
>    - This has been accomplished in the MR, but the calculation for the 
> total number of elements isn't very helpful.

I should note again, as I did previously in this same thread[0], that
BuildStream used to do exactly this.

These codepaths were discarded *because* we had two separate logging
phases, one pre-logger phase at load time and another one after the
pipeline is loaded.

The logging was since refactored a lot so that it could be initialized
much earlier so that we wouldnt need to do this, only the fine grained
logging of progress was never restored (now we only have the
START/SUCCESS messages while loading).

[...]
> * Report progress when resolving elements (convert MetaElements into 
> Elements)

This also used to exist, and we ripped it out intentionally *because*
it presented a dual standard in logging, it was not reimplemented in a
saner way since then.

If we do this, we should do it very well, or wait until the scheduler
and frontend process separation is done before doing this at all.

[...]
> * Emit status messages when fetching a junction
>    - This won't be represented with a progress indicator, but fetching 
> junctions is another case of not-readily-apparent slowdowns when 
> building.

This part would be a good step forward, but *please* do it by running
the scheduler - as discussed earlier in this thread, it's already a
problem that we go and fetch things without running the scheduler to do
so.

Fixing this in this way, will likely be completely transparent to the
scheduler and frontend process separation, as things will "just work"
in the way they were intended whenever running any tasks.

[...]
> Once this is in a good mergeable state, my next step will be to define 
> an API for plugins to report their progress, and implement progress 
> messages in the git source, since that is another point where we spend a 
> long time sitting and waiting with no feedback.

I think a better use of your time, which would be beneficial to both
this task and also to the scheduler/frontend process separation, would
be instead to start by completely untangling the frontend with the
core, and ensure that the frontend's messaging is completely driven by
Context and Stream.

For this task, it will already be important to represent Job's which
are not scheduler processes, as outlined here[0], and for the ongoing
work to split the scheduling process and frontend process in two, it
will also be crucial that the Stream present it's own representation of
a "Task", so it would be a net benefit to both objectives and it would
be work put towards a common goal.

As far as I can see, rushing forward to implement this early logging
both:

  * Potentially regresses architectural corrections we made by ensuring
    that we do *not* have any dual standards/codepaths for logging.

  * Adds more burden to the larger task of process separation by adding
    yet more codepaths to deal with in this separation, instead of
    standardizing on a common API path for logging.

Please reconsider this.

Cheers,
    -Tristan

[0]: https://mail.gnome.org/archives/buildstream-list/2019-April/msg00063.html