[BuildStream] Reconsidering BuildStream's CLI design

[tristan maat codethink co uk (Tristan Daniël Maat)]

Hi,

Considering we want to make sure we hit all CLI pain-points before 2.0
[1], James and I sat down to try and make sense of the current
incarnation and all of its use-cases. We tried to think about what
would happen if we re-designed BuildStream's CLI from scratch, taking
into account Chandan's design principles from earlier this year [2].

I'd like to report the inconsistencies we found, to see if I can start
some discussion on a broad rework for the CLI. The list will be in my
perceived priority order.

Without much further ado, here's what we found:

Implicit Actions
~~~~~~~~~~~~~~~~

Most commands can do certain things implicitly - for example, builds
will implicitly fetch sources for the elements to be built. What is done
implicitly varies between commands, which we argue might confuse the
user, and some implicit actions such as fetching or pushing may be
destructive or time consuming - in the spirit of our design principles
[2], we wanted to make an exhaustive list of these problems.

We came up with the following table to represent our current UI - read
as "if we (row heading), should we implicitly (column heading)", so
for cell 2,7, if we source track, should we implicitly source push?
Ticks (✓) mean yes, crosses (✗) no, and fs (f) signifies that the
combination should be toggle-able with a flag (with the other
character identifying the default).

+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|                  | Source Track | Artifact Pull | Source Pull  | Source Fetch |    Build     | Source Push  
| Artifact Push |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|Source Track      |      ✓       |       ✗       |      ✗       |      ✗       |      ✗       |      ✗       
|       ✗       |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|Artifact Pull     |      ✗       |       ✓       |      ✗       |      ✗       |      ✗       |      ✗       
|       ✗       |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|Source Fetch      |      ✗f      |       ✗       |      ✓       |      ✓       |      ✗       |      ✗       
|       ✗       |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|Build             |      ✗f      |       ✓       |      ✓       |      ✓       |      ✓       |      ✓       
|       ✓       |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|Artifact Push     |      ✗       |       ✗       |      ✗       |      ✗       |      ✗       |      ✗       
|       ✓       |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|Artifact Checkout |      ✗       |      ✗f       |      ✗       |      ✗       |      ✗       |      ✗       
|       ✗       |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|Source Checkout   |      ✗       |       ✗       |      ✗       |      ✓       |      ✗       |      ✗       
|       ✗       |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+
|Shell             |      ✗       |      ✗f       |      ✗       |      ✗       |      ✗       |      ✗       
|       ✗       |
+------------------+--------------+---------------+--------------+--------------+--------------+--------------+---------------+

The commands here are deliberately grouped - as it turns out, there are
conceptual pairs of commands, whose main difference is that one works on
sources and the other on artifacts.

The commands here are inconsistent. Here's a list of notable
differences:

1. Unlike `build`, `source fetch` will not implicitly push sources.
2. Implicit fetching/pushing/pulling usually do not have toggles, unlike
   tracking.
3. -> Except `artifact checkout`.
3. `source pull` is missing
4. `source push` is missing
5. Neither `shell` nor `checkout` have a flag to enable building,
   despite the flag to enable pulling.

After some discussion with Daniel and James, we figured that making
*all* implicit behavior explicitly configurable through user
configuration. We'd like to see something like this:

```yaml
implicitly:
  pull-sources: when-building
  track: no
  fetch: yes
```

Where the keys refer to the actions listed in the table above. By
default, most options should be set to "when-building", and we would
avoid doing anything implicitly in any other situation.

Here is my personal wish-list:

+------------------+---------------+
|                  |   Implicit?   |
+------------------+---------------+
|Source Track      |      no       |
+------------------+---------------+
|Artifact Pull     | when-building |
+------------------+---------------+
|Source Pull       | when-building |
+------------------+---------------+
|Source Fetch      | when-building |
+------------------+---------------+
|Build             | when-building |
+------------------+---------------+
|Source Push       | when-building |
+------------------+---------------+
|Artifact Push     | when-building |
+------------------+---------------+

This would maintain the sanctity of the build command (by giving a
third "when-building option" to allow builds to do everything they
please), while giving a much more standardized and comfortable
interface for everything else.

The commands themselves should of course override these user settings
- if asked to `bst source fetch`, BuildStream should still fetch even
if it is disabled in the configuration. It just will not be done
*implicitly*, therefore being less surprising to the user.

Rather than just implicit these options should also be
"opportunistic", in that BuildStream will do these things if it is
possible and makes sense for the command in question. That way a `bst
source pull` with opportunistic `source-push: yes` should push sources
to source servers that do not currently have the source.

The only action that should not be performed implicitly during a build
is `bst track`, because it is potentially destructive (though users
can enable it if they like).

This would also clean up the front-end code a little bit, since these
things would no longer need to be options for each command.

Networking flags need standardization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

One of the main problems in the previous set is the lack of a
comfortable way to disable fetching/pulling/pushing. This is possible
through the use of `--fetchers 0` and similar options, but I believe
the concept is a little odd from a user's point of view. It's
non-obvious that a 0 is allowed there.

Some commands have also already gained flags to enable/disable
pushing/fetching/tracking on top of these (see `bst shell --pull`),
and I expect this to continue if we don't add a generic interface for
it.

We should get a way to do this across all commands, and similarly to
the previous point add user configuration options to do so.

Dependency scopes aren't consistently allowed everywhere
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A handful of commands can specify what dependencies should be affected
by them. They all use the `--deps` flag - the options for these are
"None", "Plan", "Run", "Build", and "All".

Since we're making tables, here's another to describe which commands
support what:

+------------------+-----+-----+-----+-----+-----+
|Command           |None |Plan |Run  |Build|All  |
+------------------+-----+-----+-----+-----+-----+
|Artifact Delete   |✓    |✗    |✓    |✓    |✓    |
+------------------+-----+-----+-----+-----+-----+
|Artifact Show     |✓    |✗    |✓    |✓    |✓    |
+------------------+-----+-----+-----+-----+-----+
|Source Checkout   |✓    |✗    |✓    |✓    |✓    |
+------------------+-----+-----+-----+-----+-----+
|Artifact Pull     |✓    |✗    |✗    |✗    |✓    |
+------------------+-----+-----+-----+-----+-----+
|Source Track      |✓    |✗    |✗    |✗    |✓    |
+------------------+-----+-----+-----+-----+-----+
|Artifact Checkout |✓    |✗    |✓    |✓    |✗    |
+------------------+-----+-----+-----+-----+-----+
|Artifact Push     |✓    |✗    |✗    |✗    |✓    |
+------------------+-----+-----+-----+-----+-----+
|Build             |✗    |✓    |✗    |✗    |✓    |
+------------------+-----+-----+-----+-----+-----+
|Show              |✓    |✓    |✓    |✓    |✓    |
+------------------+-----+-----+-----+-----+-----+
|Source Fetch      |✓    |✓    |✗    |✗    |✓    |
+------------------+-----+-----+-----+-----+-----+

Most commands support slightly different things. There are two broad
groups, one that doesn't support plan, and another that doesn't
support plan, run or build, but it's hard to see a pattern here.

Something that particularly bugs James is that "plan" is a
particularly odd name for a "dep". "dep" is also an odd name for a
full length option. "plan" and "run" are also somewhat different
internally (one is a dependency scope, the other a pipeline
selection).

It might make sense to split up the two concepts, and make separate
options for them, i.e., "dependencies" (renamed) and "pipeline". We
should also rethink which commands should support what, and make the
list slightly more consistent.

`bst shell` is too inflexible
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Over time we've had several issues with `bst shell` not quite covering
people's use cases - in particular, it's hard to build projects in
"debug" mode, because we'd need to include debug applications in the
production pipeline.

The command has since gained `--sysroot`, which allows users to
specify a directory that should be used for the sandbox, `--mount`
which allows mounting a host-path.

While this solves those problems, I don't think it's ideal. Users need
to keep around additional project-related directories that will be
hard to share and use in a group.

It would be nice if we could tie `shell` back more into BuildStream
elements. What I would like to see is something like this:

    bst shell [--stage-sources ELEMENT...] ARTIFACTS...

Where `ARTIFACT` is the artifact on top of which BuildStream will
stage `SOURCES`. Since artifacts are no longer tied to specific
projects, this gives us the flexibility of using *any* sandbox
(provided it's accessible through a cache), including other or
junction projects'.

This design covers the following use cases:

1. A user wants to try out their recently built artifact. This can be
   achieved with `bst shell artifact` or `bst shell element.bst`.

2. A user wants to try building a new element and use the shell to
   "try out" a build. This is achieved by placing the relevant sources
   in the correct artifact, i.e., `bst shell --stage-sources test.bst dependency.bst`

3. A user wants to run a debugging tool on their element. This can now
   be done by supplying an arbitrary artifact - even one not usually
   included in the production image, making it easier to build these
   tools in the right context.

4. An organization wants to supply a shared "debugging" environment,
   and maintain it as a BuildStream project. This is now easily
   achieved, and can even be made simple to use with the help of
   junctions: `bst shell --stage-sources test.bst org-junction.bst:test.bst`.


I believe the pieces for this are all in place in some fashion, so
this should be a much less impactful change than it seems.

The use case of staging a non-BuildStreamed sandbox (i.e., a normal
sysroot) would not be as easily achievable, but I believe that this is
just a band-aid we applied after realizing that `bst shell` didn't
always do what we needed.

In exchange, this would make use cases 3 and 4 much more comfortable
to use, since users wouldn't need to manually maintain debugging
sysroots, or manually check them out whenever they want to run
something.

`bst source` and `bst artifact` should work outside project directories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There are now use cases for this, most importantly deleting artifacts
from a cache without needing to be in a BuildStream project
directory.

Of course this does not make sense for *all* their subcommands, the
list that I would propose is:

- checkout
- delete
- list-contents
- log
- pull
- push
- show

Assuming that these are implemented for sources. Notably, `fetch` and
`track` are excluded.

"Track" is an odd word
~~~~~~~~~~~~~~~~~~~~~~

I like interfaces that don't require a manual for the most part, where
commands and options are fairly self-evident and can be learned just
by browsing and looking at options that look somewhat like what I want
to do. I think BuildStream is mostly there.

However, if I, as a new user, wanted to update my dependencies, I
would check `bst source --help` - and probably overlook `track`,
because the name is non-obvious. Maybe this should be renamed to
"update"?

It's minor, and may be too far down the line at this point, but if that
is not changed now we won't get the chance to until BuildStream 3.0 -
which hopefully is still a while out.

---------

That wraps up the changes I wanted to suggest for now

Thanks,

Tristan Maat

[1]: https://gitlab.com/BuildStream/buildstream/issues/1068
[2]: https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html
[3]: https://gitlab.com/BuildStream/buildstream/issues/685
[4]: https://gitlab.com/BuildStream/buildstream/issues/1065
[5]: https://gitlab.com/BuildStream/buildstream/merge_requests/1561