Re: [BuildStream] Proposal: Allowing download-only sources to work on local files

[Tristan Daniël Maat <tristan maat codethink co uk>]

Hi,

For reference, I've wanted to see a feature like this since my second 
week on the project when I realized
that the test suite needed to access local tarballs, so I'm very in 
favor of this, albeit not in its current form :)

> > - All source plugins which operate on files are unified as subclasses of
> >    a single base class, say FileBasedSource
> > - This base class handles the 'url' and 'ref' keys of the source config
> > - First it checks whether 'url' is a fully-qualified URL or is just a
> >    relative path. If it's the former, it is fetched as necessary and
> >    stored in the source cache
> > - If the URL is a relative path, specifying a ref is optional. If it is
> >    given and is different from what is calculated, an error is throw
>
> This seems like the path of smallest disruption to existing users I 
> would be tempted by not letting local
> not have a ref unless specified in project.conf but dose that sound 
> too disruptive to others?

Unlike Will I'm not too happy about this, we'd need to classify whether 
a path is a URI or a relative path.

This is difficult, because a URI could be a perfectly valid path name, 
albeit a rather horrible one. While
this isn't the most realistic use case, I'd much rather see a way to 
have both a `url` and `path`, as described
for `remote` and `local` sources, simply to stay far away from using 
heuristics to decide anything - it
feels wrong to have this ambiguity when we want precise definitions of 
inputs.

The simplest way to implement this would be to have an implementation 
for a new `path` key which
simply performs the old operation, except with 
`file:///path-to-project/<path>` to prevent any disruption.

> > The 'local' and 'remote' sources could then also be unified to some
> > kind of 'copy' source, as 'copy' would act as either one depending on
> > whether a local path or full URL was given.
> >
> > If desirable, the 'url' key could be split into mutually-exclusive
> > 'url' and 'path' keys which would decide the behaviour.
>
> Having two seems more explicit which seems more sensible but I don't 
> have strong feelings here. With `path` for `relative to project` then 
> url/uri as it works now.
>
> For the single url key, would it work like: `uri: file://../bob.txt` 
> for relative vs absolute `uri: file:///bob.txt`, this seems workable 
> to me.
Just for reference here, the URI RFC explicitly disallows `./..` as part 
of path names. Again, I'd rather we don't use a single key, but
instead stick to path/url, simply to stay consistent with the standards 
we're using - if only to make parsing simpler on our end and
stay somewhat inter-operable with other tools.

Regards,

Tristan