Re: [BuildStream] Proposal: bst source delete command

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Chandan,

> On Feb 22, 2020, at 6:49 AM, Chandan Singh <chandan chandansingh net> wrote:
>
> Hi all,
>
> BuildStream currently offers no way of systematically deleting locally cached
> sources. I think it should however do so - not only to maintain symmetry with
> the `artifact delete` command but also because sources can be substantial in
> size in many cases.
>
> In this message, I propose to add a new `source delete` command.

Not really against this, but I’d like to point out that this is not a usage pattern I’d want to require of 
normal users. The artifact delete command was created mostly for the convenience of developers, e.g to be 
able to easily rerun tasks in their plugin while developing it.

The user on the other hand should simply be given the convenience of BuildStream respecting the users decided 
cache quotas, and never be pestered to manually type a command or asked to know which things they’d like to 
delete.

From this perspective, I’d hope to see this as a power user feature (which is fine to have I think).

> Unlike its `artifact delete` counterpart, this is slightly more nuanced as we
> need to delete the sources from two places:
>
> 1. From the local CAS cache
> 2. From the plugin's cache directories
>
> (1) is simple, but (2) would require cooperation from source plugins. To that
> end, I propose to add a new abstract method for source plugins, like
> `Source.delete()`.

I’m not convinced about this part.

This will require some API change support from the plugin, but instead of adding more to plugins, I feel like 
we might get away with being more strict with the plugin’s cache directories.

If we instead decide that plugins need to ask for a directory while specifying the name of the source which 
can later be given to `bst source delete`, then BuildStream retains ownership of the directory and do what it 
pleases with it later, without asking the plugin to do anything.

> Plugins can choose to do nothing if they don't maintain an extra local cache,
> like local and workspace sources.
>
> Plugins that do maintain a local cache can implement this method and delete the
> corresponding directories from their cache directories. For example, the `tar`
> plugin would delete something like `~/.cache/buildstream/tar/URL_FOR_TARBALL`.
>
> Open questions
> --------------
>
> 1. Since source plugins do not know about how many times a a given source is
>   used, they can't do reference counting of sources. So, imagine a scenario
>   where elements A and B both use a git repository G. Using this approach,
>   `bst source delete A` would end up deleting it, as we have no way of knowing
>   that B is also using it. This may not be too bad but the user may have
>   expected the source to persist if another element is still using it.

This is a good question, what exactly is `A` in `bst source delete A` ?

I wasn’t expecting that we’d be passing element names here, but how can we address sources directly ?

Cheers,
    -Tristan

>   I think this is probably fine but we should make this quite clear in the
>   relevant documentation.
>
> 2. What to do with unreachable sources? How do we delete them? It is not
>   inconceivable to imagine that certain URLs are no longer referenced. But,
>   again, we don't have any way of ascertaining that.
>
> I don't have answers to either of these questions so I'd be curious to hear if
> you have any ideas. Nevertheless, I don't think they are blockers for this
> feature and we can put them on the back burner if can't find a solution for them
> right now.
>
> Let me know what you think.
>
> Thanks!
> Chandan
> _______________________________________________
> buildstream-list mailing list
> buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list