Re: [BuildStream] API proposal - enable iterating through all elements

[Chandan Singh <chandan chandansingh net>]

Hi Douglas,

Many thanks for this write-up. We've had similar discussions here and there so
I am really pleased to see a dedicated thread for it.

In the past, people have (casually) expressed to desire to have access
to some of BuildStream API, that is not necessarily for plugin authors, but for
people trying to build other tools on top of BuildStream. As with all public
API, we need to be careful of how and what we expose.

Some more comments inline.

On Mon, Jan 13, 2020 at 3:57 PM Douglas Winship via buildstream-list
<buildstream-list gnome org> wrote:
> Hi all
>
> I'd like to propose making some more elements of the API external / public facing. We're trying to automate 
> some tasks for monitoring source URLs and mirror URLs, and we haven't found a way to do it without using 
> internal API features.

I appreciate this use case and what you are trying to do. I was trying to do a
similar thing in the past where I wanted to ensure that all elements were using
_some_ alias for their URLs. That has similar requirements so I'm glad to see
other related use cases as well.

I also think that you are right in saying that the current public API can't
solve this fully.

<snip>

> A much more direct solution, that does what we actually want, would be to iterate over all of the files in 
> the elements directory, and to be able to work from there. But we don't know of any way to do that without 
> using protected parts of the API.

Using the API (public or private) is surely one way to go about doing this. I
wonder if the same can also be achieved via the CLI. We don't support it at the
moment, but maybe that would be another way of solving it. Consider this
half-baked proposal:

1. There is a bst command that you can run to get the list of all aliases
  that are accessible for a given element.

2. There is a bst command that can output the field called "url" for all
   elements in a project in a known format.

While we support some format options for `bst show` (see help text), we don't
have anything that can output source fields just yet. But if we had the ability
to print those and all available aliases, do you think that would be useful for
your use case?

> I've got an example script, showing the kind of thing I'd like to be able to do. The example uses 
> app.initialized() from the _frontend submdodule. The example is on the FreeDesktop-SDK project: 
> https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/DouglasWinship/mirror_url_manifest/utils/mirror-management-scripts/url_manifest.py
>
>
> Wishlist: things we think we'd need for this use case:
> - Essential: A function for source objects, that that produces mirror urls (like 'translate_url' does for 
> the main source URL).
>   - (OR access to the mirror-aliases, so we can do our own substitutions)

This bit may not be as straightforward as it seems, since it's not BuildStream
itself that calls the `translate_url` method. The plugins do. And BuildStream
has no way of knowing where that URL comes from. For all BuildStream knows,
the plugin could be passing it a hardcoded url each time.

So, even if `translate_url` method becomes a public static function,
BuildStream still
won't know which URLs to translate. Of course, you as the author of this tool,
may know which plugins you use and what do they call their URL fields.

My point is that I'm not sure if it's feasible to have a generic method on
BuildStream side that will produce mirror URLs without having knowledge about
what plugins are being used.

Providing access to all alias definitions seems less controversial and more
feasible to me. We can even expose that as a CLI format option as I proposed
above.

> - Essential: The ability to somehow instantiate an 'element' object, given a '.bst' filename. (The current 
> API only seems to allow this when writing a plugin. We'd like to be able to do it using something like the 
> current app or stream methods, as in the example script).

This is the more tricky part :) I don't claim to have all the answers here but
essentially this is about making parts of the API public, other than what we
have for plugin authors. As such, I think this warrants a larger discussion of
what bits we want to expose and how.

Elements, by definition, only make sense in the context of project.
So, we are implicitly asking for a public way to instantiate Project
objects as well. Which means that we have to make X, Y & Z public, and
so on. My point being that this will require some careful thought.

> - Desirable: a function or an attribute for sources, that gives the alias.

Can you please clarify what you mean here? Do you wish to get the raw value of
the alias?

> - Desirable: access to the value stored in project.element_path (so that the script doesn't have to assume 
> a directory called 'elements').

Assuming that we open up some sort of API, definitely +1 in that case. If not,
I again wonder if the format options of `bst show` can be enhanced to satisfy
this requirement as well?

Thanks!
Chandan