Hey Everyone,
TLDR: The plugin migration outside of BuildStream is breaking both
some plugins and BuildStream, I'm proposing to have more thorough code reviews
on those changes in addition to changes on how tests are run.
---
# Why is testing plugins with BuildStream important
This might seem redundant but I want my message to be clear:
Now that we move plugins out of BuildStream core, we get less test coverage
on BuildStream itself, since those plugins might be testing edge cases.
This could lead to BuildStream becoming more fragile, with more bugs, and
therefore make it less usable.
We also need to now ensure that widely used plugins, once out of the core, stay
functional for our users, which we had for free before. Now we need to be
cautious about them.
While we started moving plugins out of core, some errors went unnoticed, for
example, the `ostree` plugin, once moved was actually never tested against
BuildStream. Even now, _no_ external source plugin is tested against the
standardized source tests, because they are not registered correctly.
This can lead to breakages once we start moving more complex plugins out of
core, and I really want to avoid that.
# Problems added with the migration:
- Plugins are now installed by default when running BuildStream tests, which
means our tests are not standalone anymore.
- The standardized source tests where not checked to be run,
and are currently not running (they never were).
- BuildStream is now tested against a fixed version of the plugins that is
not updated frequently.
I want therefore to update how we deal with processes and testing external
plugins.
# On the process side:
I would like us to consider having more thorough code reviews and checks as
we do on BuildStream to avoid such problems, and ensure the process is visible.
# For testing:
This is my take on how to ensure that BuildStream doesn't break once we have
less plugins to test against.
## On BuildStream side:
External plugins that want to be tested as part of BuildStream CI should define a
methods `{package_name}.testutils.register_sources()` (We currently don't have
standardized elements tests).
This methods would call all the `register_repo_kind` necessary.
BuildStream would have a new test target (`-m sources`) which would run the
standardized source tests against each sources that declares an entrypoint in
the current environment.
Tests would be run on BuildStream CI in parallel, once against a fixed tag of
the plugins, once against their master version. The master version being
allowed to fail.
BuildStream tests that are not part of the standardized source tests should
_not_ be allowed to require external plugins and should be entirely standalone.
Integration tests are an exception to that and should be moved to their own
environment.
This would lead to three types of (tox) environments:
- pyXX
- pyXX-integration
- pyXX-plugins
## On the plugins side:
The plugins should run tests against both a pinned version of BuildStream and
against master to ensure consistency. The tests against the BuildStream master
branch being allowed to fail.
## Breaking changes:
When doing breaking changes (pre-2.0, after there won't be any so that is not a
problem), the plugins should be modified first and merged to their master
_while being broken_.
# When moving a plugin:
1) Rewrite all tests that are testing internals of BuildStream that depend
on the plugin to be removed
2) Copy the plugin with its tests to bst-plugins-experimental
At this point:
- *No* code change must happen, in either repositories
- Tests must be moved alongside the plugin
- Source plugins should be registered in `register_sources()` in the
plugins.
# Unanswered questions
Those questions are here merely for bookkeeping and I don't intend in resolving
all of them in this thread.
- How do we get standardized element tests?
I volunteer myself to start updating the infrastructure and the tests next week.
Please let me know if you see any issues with the plan.
Benjamin
_______________________________________________
buildstream-list mailing list
buildstream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list
