GNOME.org
 

Re: [BuildStream] bst artifact show - PROPOSAL

Hello list,

This post is in response to Ben's proposal of the `bst artifact show`
command [0] and the replies from myself, Tristan and Sander [1].

TL;DR
1. This command would be super useful and it will give us the ability to
   be able to determine whether an artifact is cached remotely.
2. I mentioned in the thread that our current artifact subcommands
   *only* deal with the local cache. I think it's *very* important that
   we remain consistent, so, `bst artifact show` (with no options) will
   show us what is cached locally. A `--remote` option or `all-remotes`
   option will be required if we want to know whether an artifact is
   cached remotely.
       - Perhaps we could consider allowing for remotes to be declared
         with an alias, therefore making the `--remote` option a bit
         easier to use. Although, this should be an extension and not
         block the work.
3. As Tristan suggested, the "local mode" of this function will probably
   be what we had in mind for `bst artifact list`.
4. The command will support artifact refs, glob expressions and element
   names.
5. We should be able to format the output.

As I'm going to pick up this work, I will summarise how I think the
command should work (taking into account all that has been discussed).
Personally, I feel like the best way to do this is through examples,
so we'll go from there!

# bst artifact show and the local cache

Show whether or not an artifact is cached locally, or show *which*
artifacts are cached locally.

The argument of this function will be a list of targets, each target can
be either:
1. A complete artifact ref ($project/$element/$cache_key)
2. An element name
3. A glob expression for artifacts. For example, $project_name/* for
   all artifacts in a project.

Our current `bst artifact log` and `bst artifact delete` commands
already support this.

Now, for each target type, I think the user should see something like
this:

1. For a complete artifact ref
$ bst artifact show foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa [--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Resolving cached state [00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state cached foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 


Or, if the artifact is not cached this will be special cased behaviour
so the user will see:
$ bst artifact show foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa [--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Resolving cached state [00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa is not cached
2. For an element name, we will *list* all of the artifacts cached for that element. 


    $ bst artifact show bar.bst
[--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Resolving cached state [00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state cached foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa cached foo/bar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb cached foo/bar/cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc 
          ...
cached foo/bar/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz 


Or a message saying that nothing is cached for that element.
3. For a glob expression, we will list all cached artifacts which match the glob expression 


    $ bst artifact show foo/*
[--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Resolving cached state [00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state cached foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa cached foo/bar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb cached foo/bar/cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc 
          ...
cached foo/bar/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz cached foo/pony/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa cached foo/pony/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 
          ...
cached foo/pony/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
Or a message saying that nothing which matches the glob expression is cached. 


# bst artifact show with remotes

Here, I can imagine two main use cases:

1. Whether an artifact is cached in a *specific* remote
2. Whether an artifact is cached within any of the *available* remotes

I think we can achieve 1. with a `--remote` flag (to be consistent with
push/pull) and 2. with an `--all-remotes` flag.

Here is what I'd imagine 1. would look like,

1. For a specific artifact ref:
$ bst artifact show --remote http://localhost:11001 foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa [--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Initializing remote caches [00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches [--:--:--][ ][ main:core activity ] START Resolving cached state [00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state available foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa http://localhost:11001 


Where we use third column to show the remote.

Or, if it's not cached:
$ bst artifact show --remote http://localhost:11001 foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa [--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Initializing remote caches [00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches [--:--:--][ ][ main:core activity ] START Resolving cached state [00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa is not available in: http://localhost:11001 


2. For an element name:


    $ bst artifact show --remote http://localhost:11001 bar.bst
[--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Initializing remote caches [00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches [--:--:--][ ][ main:core activity ] START Resolving cached state [00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state available foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa http://localhost:11001 available foo/bar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb http://localhost:11001 available foo/bar/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz http://localhost:11001 


3. For a glob expression:

You get the idea...


Now, for the `--all-remotes` flag, we would have this command list each
remote which contains the specified artifact:
$ bst artifact show --all-remotes foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa [--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Initializing remote caches [00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches [--:--:--][ ][ main:core activity ] START Resolving cached state [00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state available foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa http://localhost:11001 available foo/bar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa http://localhost:8080 


For element names (which effectively acts as a glob expression) and glob
expressions, we will determine a list of refs, and then for each ref
determine which of the remotes they are in and then list the result.

To be efficient here, I think that we should query a remote, check all
of our refs against what's in the remote, cache the result and then
move to the next remote. We should avoid querying remotes more than
once.


# Possible extensions

Allow for remotes to be defined with an alias. This would make it easier
for the user, rather than having to use the url and port, we just
use the alias. This will also make the UI look nicer where, if
available, we show the alias name rather than the url/port.

-----

I've done my best to try and cover what I think all the use cases are
and to address what was previously discussed in the older threads.
However, there is a good chance I've overlooked some things so please
respond with your opinions/improvements/objects etc :)

As this looks like it's going to be a big new feature and I would like
to have us all in agreement before I proceed with the work.

Thanks!
James
[0] https://mail.gnome.org/archives/buildstream-list/2019-May/msg00045.html [1] https://mail.gnome.org/archives/buildstream-list/2019-June/msg00000.html
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]