Hi all
I'd like to propose making some more elements of the API external / public facing. We're trying to automate some tasks for monitoring source URLs and mirror URLs, and we haven't found a way to do it without using internal API features.
In our case, the goal is to iterate over all the different elements in a project, and generate a manifest of all the different source URLs. The manifest should be able to pull out the 'raw' URL, the alias, the processed source URL, and any processed mirror URLs. (We will then use the manifest in scripts that validate the source URLS. for example to ensure that every source URL an alias, and has a mirror defined.)
(e.g.
element: 'llvm.bst'
alias: 'github:',
raw url: 'github:llvm/llvm-project.git',
source URL: 'https://github.com/llvm-project.git',
mirror URL: 'https://gitlab.com/freedesktop-sdk/mirrors/github/llvm/llvm-project.git')
We'd like to be able to do this all using buildstream functions, but there's a number of things that we can't see how to do without calling the internal parts of the API.
The simple part:
If we have an instance of the "element" class, we can extract a list of sources, and get the raw URL directly. From there we can get the source URL using the translate_url function, all with the external API. But there's no similar external function for getting mirror URLs or the alias. Admitedly, it's easy enough to find the alias just using string-parsing methods, but the mirror URLs are another matter.
The bigger issue:
We don't want to use a plugin.
A plugin doesn't seem like the right way to approach this. On a conceptual level, this manifest isn't like an element we want to build. We don't want it to be included in any of our final outputs, and it's not about inspecting artefacts or software. It's more of a utility for code review.
Even more important is the practical level: If we implemented this as a plugin, then the plugin could only operate on one element and its dependencies. We'd have to create a new element for the manifest, and make sure that every other element in the project was somewhere in its dependency tree. The freedesktop-sdk project has several different final outputs, and there's no one existing element that necessarily depends on everything else in the project. We could create one, but that adds extra maintainence overhead, and we couldn't easily tell if we missed anything.
A much more direct solution, that does what we actually want, would be to iterate over all of the files in the elements directory, and to be able to work from there. But we don't know of any way to do that without using protected parts of the API.
I've got an example script, showing the kind of thing I'd like to be able to do. The example uses app.initialized() from the _frontend submdodule. The example is on the FreeDesktop-SDK project: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/DouglasWinship/mirror_url_manifest/utils/mirror-management-scripts/url_manifest.py
Wishlist: things we think we'd need for this use case:
- Essential: A function for source objects, that that produces mirror urls (like 'translate_url' does for the main source URL).
- (OR access to the mirror-aliases, so we can do our own substitutions)
- Essential: The ability to somehow instantiate an 'element' object, given a '.bst' filename. (The current API only seems to allow this when writing a plugin. We'd like to be able to do it using something like the current app or stream methods, as in the example script).
- Desirable: a function or an attribute for sources, that gives the alias.
- Desirable: access to the value stored in project.element_path (so that the script doesn't have to assume a directory called 'elements').
|
