[BuildStream] Proposal: Allowing downloadable file plugins to work on local files

[Tom Mewett <tom mewett codethink co uk>]

Hi list,

As discussed in #1047, it seems it would be useful to allow plugins
which are currently remote-only to accept local files in the project.

Various suggestions were made for the behaviour of this, and some work
was started to implement it, in !1423. I will outline an idea which I
think meets all the requirements mentioned, while unifying some
behaviour across source plugins.

1.
The 'url' option of current plugins derived from DownloadableFileSource
is extended to accept project-relative paths, with no scheme. These
paths are transformed internally into, and henceforth treated as,
absolute file: URLs.

2.
If a file: URL is given, the plugin behaviour changes to be similar to
that of the local source:

* fetch does nothing, except throwing an error if the file doesn't exist
* track does nothing
* the source is always reported as cached
* _get_mirror_file (to be renamed?) reports the absolute path to the
  given file

Even without the relative path feature (1), this provides benefits such
avoiding needlessly copying the file into the source cache.

The intention is to abstract from the subclasses of
DownloadableFileSource whether the file to operate on and stage, given
by _get_mirror_file, is local or remote.

3.
With this, the 'patch' source could also be brought under this
consistent API. Further, it may be sensible to actually merge the
'local' and 'remote' sources into one 'copy' source, as their behaviour
would become effectively identical.

DownloadableFileSource ought to be renamed too.

---

One area I am unsure about is the behaviour of refs. With my current
suggestions, when using local files the 'ref' option is completely
ignored and is hence optional. I would be interested in whether anyone
has a use case for specifying refs for local files. (Perhaps test cases?)

I'm very happy to accept suggestions and criticism on this proposal;
please let me know :) (I hope also that this isn't too minor of an
issue to warrant a list message... still getting the hang of this)

Thanks,
Tom

[#1047] https://gitlab.com/BuildStream/buildstream/issues/1047
[!1423] https://gitlab.com/BuildStream/buildstream/merge_requests/1423