Re: Package manager integration with BuildStream

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Mon, 2018-04-30 at 15:25 +0000, Chandan Singh wrote:
> Hi Tristan,
>
> Thanks for the quick reply! I agree that I might have exaggerated some of the
> concerns but I was just trying to point out the potential issues.
>
> >   o Either we say that the meaning of the 'directory' configuration
> >     for SourceTransforms is the CWD for it to "do it's thing"
> >
> >   o Or, we have additional configuration for it to "do it's thing"
>
> This is probably going into implementation detail territory, but I am of the
> opinion is that an additional configuration might be more clear and explicit
> I would expect the `directory` configuration to tell where the output of
> SourceTransform should go (with sensible defaults). So if we add an additional
> configuration, we can continue using `directory` for that purpose and the new
> configuration can be used to override the default value for the directory
> where it needs to "do its thing". I'm expecting that implementations of
> SourceTransform would come with sensible defaults so it shouldn't be
> something that you usually need to override.

I understand and agree that using the "directory" configuration is
confusing, it's not symmetric with how Source uses it.

I am however of the opinion that there is not any specific place where
"the output of the SourceTransform should go", I find this rather
ambiguous and implementation dependent.

On the other hand, I *do* think that "where the SourceTransform does
its work" is something which can be generalized as a common property of
such plugins.

I think that here, I may just be emphasizing the same point that you
are also making (but in case I am not, further clarifications follow
below...).

For disambiguation, an alternative approach could be made with a bit
more boiler plate, i.e. we could have a SourceBase class from which we
derive both Source and SourceTransform classes, such that
SourceTransform does not carry the "directory" by itself, and could
define a "workdir" or such instead.

> > >   Extending this problem, what happens if there are multiple sources
> > >   with different destination directories; how do we communicate those
> > >   to the final SourceTransform source?
>
> > This is pretty much a domain specific problem, which can be solved in
> > ways which are appropriate to the SourceTransform implementation I
> > think.
> >
> > Rather, what examples do you have in mind here ?
>
> This is probably not a common use-case but I was imagining the second case that
> you mentioned, i.e. "staging a single build with multiple different
> subdirectories with different languages"
>
> > So stage 3 different SourceTransform elements after it, and tell
>   each one of them separately where to go and do their thing.
>
> Yes, that's what I was trying to get at, i.e. there has to be a way to tell
> each one of them where to "do their thing". But, as discussed above, this can
> be a configuration option so this shouldn't be an issue.
>
>
> One final thing that I forgot to mention in my last message - it may be just
> me but I found the name "SourceTransform" a bit confusing. Without any context,
> I would have imagined a source that transforms files staged by other sources
> (similar to how `patch` works). Since the intent is not to transform files
> already staged by other sources, would something like "DependencyManager" be a
> more appropriate name for this kind of source or would that make the scope too
> narrow for this source kind?

I don't quite agree that the intent is not to modify existing files,
rather the intent is to modify a directory; this might very well end up
intentionally modifying files (or overwriting them) in the transformed
directory.

For instance, it is typical for a release tarball which uses rust to
include a Cargo.lock file; I would imagine that it is well within the
rights of the SourceTransform to override this when tracking. Whether
that sort of thing is optional behavior or not, the point should remain
that from the point of view of the core, we are permissive and we allow
the SourceTransform to do whatever it wants.

Circling back to the previous point, that there is no "location where
the result is to be staged":

This basically means that different SourceTransform plugins might do
totally different things to the Sources they transform; it might
create a directory, it might also create a Cargo.lock file... it might
also create a cargo config file which informs the rust tool chain where
to obtain sources... and all of these separate things might have to be
configurable.

Maybe SourceTransform is not the very best name possible, but I'm not
fond of naming the class after the specialized use case it was
originally created for, but would prefer to name it after what it is
capable of doing in a more general sense.

In other words: There may very well be other use cases we haven't
thought of which this will be useful for, beyond automating the
fetching and tracking of dependencies.

> I have also created https://gitlab.com/BuildStream/buildstream/issues/381 on
> GitLab to track the implementation of SourceTransform plugin as it seems like
> there are not many pending questions at this point.

Thanks !

Cheers,
    -Tristan