Re: [BuildStream] How to catch regressions in buildstream's interactive interface

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Mon, 2018-10-15 at 17:59 +0100, Jonathan Maw via BuildStream-list wrote:
> Hi,
>
> I am working on an issue that involves changing how buildstream works 
> when you use buildstream in interactive mode, fail a build, and launch 
> the shell. While it's mostly the same codepath as `bst shell --build`, I 
> am concerned that my ability to test its workings are extremely manual.
>
> The solutions I can think of are:
> * A checklist of behaviour that can only be manually checked? Bad, 
> because it'd lead to an ever-expanding list of things to check as time 
> goes on.
> * Some kind of test harness that monitors stdout/stderr and can write 
> commands to stdin? Very hard to write something general-purpose, and 
> since it's not using machine-readable output, prone to behaving in 
> unexpected ways.
>
> Is this a goal worth pursuing? Does anyone know any other solutions?

Definitely the latter yes. We need a test harness which allows
interactive communication with BuildStream.

This is a significant hole in our current CI and is definitely worth
pursuing :)

Cheers,
    -Tristan