Re: [BuildStream] Proposal: Allowing download-only sources to work on local files

[Tom Mewett <tom mewett codethink co uk>]

Hi, thanks both for your thoughts.


On Mon, 2 Dec 2019 12:42:37 +0000
William Salmon via buildstream-list <buildstream-list gnome org> wrote:

> > Another advantage would be that local files imported into the project
> > can be given refs, meaning that they would not need to be present to
> > compute cache keys of depending elements. (This is not possible with
> > the current 'local' source.)
>
> Whether or not they are present is useful, especially in CI but for big 
> files the real advantage for me is that you do not have to go through 
> the CPU and IO intensive action of hashing the file for every invocation 
> of bst, the act of running show on a built top level element could 
> require the Hashing of many large files and end up taking a large chunk 
> of the time required to run show.

I don't think this is plausible. What I think you're suggesting is that
BuildStream trust the ref is correct without actually checking - if it
did this then it would be possible to alter the local files and hence
change an element without affecting its cache key.


On Mon, 2 Dec 2019 14:13:16 +0000
Tristan Daniël Maat via buildstream-list <buildstream-list gnome org>
wrote:

> Unlike Will I'm not too happy about this, we'd need to classify whether 
> a path is a URI or a relative path.

I hadn't considered this. Using separate 'url' and 'path' keys is then
a good option, I think, and fits with how the relevant plugins
currently work.

Being picky, I would personally try to avoid having two
mutually-exclusive keys for the same thing; perhaps the proposed single
'url' key could be named something less confusing (I'd choose
'location' but that's already taken; 'from' ?) and there could be a
'local:' scheme for use with local files. That might be an uglier
change, however.

  - kind: zip
    from: local:files/some-archive.zip

> The simplest way to implement this would be to have an implementation 
> for a new `path` key which
> simply performs the old operation, except with 
> `file:///path-to-project/<path>` to prevent any disruption.

This might be simplest in terms of amount of change but I don't think
it's good from a design POV, since it just adds to the current
inconsistency instead of resolving it. 

-- 
Tom Mewett <tom mewett codethink co uk>