Re: Proposal for Remote Execution

[Jim MacArthur <jim macarthur codethink co uk>]

On 11/04/18 21:37, Jürg Billeter wrote:
> Virtual File System API
> ~~~~~~~~~~~~~~~~~~~~~~~
> As a second phase I'm proposing to introduce a virtual file system API that
> both BuildStream core and element plugins can use in place of path-based
> file system operations provided by the OS. The goal is to allow transparent
> and efficient manipulation of Merkle trees.
>
> The API consists of an abstract Directory class that supports creating
> directories, copying/renaming/moving files and whole directory trees, and
> importing files from a local path. See also the existing `utils` functions
> `link_files` and `copy_files`, equivalent operations need to be supported.
>
> Proposed steps with additional details:
> * Create abstract Directory class.
> * Implement regular OS file system backend.
> * Add get_virtual_directory() to Sandbox class returning a Directory object,
>    still using regular OS file system backend for now.
> * Add boolean class variable BST_VIRTUAL_DIRECTORY for Element plugins to
>    indicate that they only use get_virtual_directory() instead of
>    get_directory(). Add error to Sandbox class if get_directory() is used
>    even though BST_VIRTUAL_DIRECTORY is set.
> * Port users of Sandbox.get_directory() to get_virtual_directory():
>    - Element class
>    - ScriptElement class
>    - Compose plugin
>    - Import plugin
>    - Stack plugin
How do you think we should put sources into this virtual file system 
before the FUSE is implemented? Source plugins such as `tar` require a 
directory to pass to the Python tarfile module, so we can either expose 
the underlying file system directory to the plugin (not ideal) or use 
tarfile to extract to a temporary directory and then copy/link it into 
the virtual file system, which might be a performance penalty. Any thoughts?

Jim