Re: [BuildStream] Proposal: Allowing downloadable file plugins to work on local files

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Tom,

Thanks for the writeup :)

On Thu, 2019-08-15 at 17:30 +0100, Tom Mewett via buildstream-list wrote:
> Hi list,
>
> As discussed in #1047, it seems it would be useful to allow plugins
> which are currently remote-only to accept local files in the project.
>
> Various suggestions were made for the behaviour of this, and some work
> was started to implement it, in !1423. I will outline an idea which I
> think meets all the requirements mentioned, while unifying some
> behaviour across source plugins.
>
> 1.
> The 'url' option of current plugins derived from DownloadableFileSource
> is extended to accept project-relative paths, with no scheme. These
> paths are transformed internally into, and henceforth treated as,
> absolute file: URLs.

So I wonder now, is there a reason to limit the impact of this change
to only DownloadableFileSource ?

It seems to me that this pathing kind of feature fits nicely into what
`Source.translate_url()` does, perhaps we could allow expansion to
project relative "file://..." URIs across the board for all plugins ?

> 2.
> If a file: URL is given, the plugin behaviour changes to be similar to
> that of the local source:

Nit: The `local` source uses `path:` here, would probably be better to
use the same word for consistency...

[...]
> One area I am unsure about is the behaviour of refs. With my current
> suggestions, when using local files the 'ref' option is completely
> ignored and is hence optional.

Just pointing out that if there are cases where refs can be set but are
not used - specifying them in the project should probably raise an
error, or a warning that can be configured fatal.

> I would be interested in whether anyone
> has a use case for specifying refs for local files. (Perhaps test cases?)

The `refs` are basically for indexing content at `url`, or for telling
us more precisely what the content is, in the case of downloadable
files.

The cache keys for local Sources are not any weaker than for Sources
which use refs, they are just resolved completely dynamically and on
the fly, without the need of handling a ref.

Take this use case for example:

  o You're in an edit/compile/test cycle where you are trying to get
    something to boot properly, or you're debugging something in the
    boot sequence.

  o You're editing the `/init` file, the first script that runs inside
    your initrd.

Currently, if you modify the `/init` script locally then that will
cause your initrd to magically get a new cache key - if you undo your
changes to `/init` then you get the old cache key (and probably already
cached artifact).

If local sources had refs, then instead we'd have the same cache key,
and an error informing us that what is found in the given local
directory does not match the checksum which is known by the element.bst
file (essentially indicating a problem with the data, or that we need
to `bst source track` to update our ref).

It may be that there are cases a user would want to opt into using refs
for local files, and it appears your proposal (1) would allow this.

In general, I'm pretty happy with the default stance that "project
paths are a part of the input". As such we resolve their cache keys
dynamically (we don't need to police the input with input and cross
check it) - this seems to be the most convenient as a default.

Cheers,
    -Tristan