[BuildStream] BuildStream tests and external plugins

[Benjamin Schubert <contact benschubert me>]

Hey everyone,

Apologies for those receiving the message twice. It apparently didn't arrive

to the ML archive...

TLDR: I propose to add a `@pytest.mark.plugins` mark on every test requiring
a plugin that is/will be external and run tests in sequence: first the ones
that are entirely self contained ("Buildstream core") then the ones with
other plugins ("BuildStream integration")

---

As we are moving forward to BuildStream 2.0, we will need to move most of the
plugins still in core outside.

Most of the still remaining plugins are there because we depend on them in the
test suite and we have no clear story on how to migrate them yet.

For a reminder, this is how the tests work currently:

We have three types of tests:

- Core tests, don't depend on a sandbox nor on external plugins
- Integration tests, don't require external plugins but need a sandbox.
- Source plugin tests, are run against a set of internal and external plugins
  and are allowed sandbox access.

This is not ideal if we want to separate more plugins out of the core.

This is what I propose in order to allow us removing the plugins from the core
while still keeping a thorough test suite:

- Every test that tests _the plugin functionality_ and not a core functionality
  should move with the plugin. It is not BuildStream core's responsibility of
  running those tests.

- We mark every test using external plugins as `@pytest.mark.plugins`.
  That allows us to know immediately if a test requires external plugins or not.

- We run those tests at the same moment as we run source tests (which already)
  require external plugins.

All generic existing source tests would get this marker.

This has a few advantages:

- It is clear which tests require an external system
- It mixes well with the `integration` mark we already currently have
- It is minimally invasive

And disadvantages:

- We need to be careful in only keeping tests that actually test a _core_
  functionality, otherwise we will end up with tests for external plugins in
  core, and no tests outside.


How would we test those in practice?

- We would be able to run the tests with `pytest -m plugins`. Or not run them:
  `pytest -m 'not plugins'`.

- We would have a tox environment installing the necessary external plugins
  (e.g. Git, tar) and running pytest correctly (e.g `tox -e py37-plugins`).


Does that seem an acceptable plan? Am I missing some edge cases?

Cheers,

Ben