[BuildStream] Guidance on unit tests

Hello list,

I have opened an MR to add guidance on unit tests to the contributing guide:

I’m posting it here too so that it gets a decent airing, it would be significant to all contributors.

One aim of mine here is to decrease uncertainty around whether unit tests are welcome in the project.

Another is to establish some common ground, before later proposing more pro-unit-test changes.

Changes proposed in this merge request:

State that exercising things via the cli is to be preferred. Perhaps it should also say the public API.

State that unit tests are good where they are robust to unrelated changes made to internals.

The current wording is:

- - -

You should first aim to write tests that exercise your changes from the cli.

This is so that the testing is end-to-end, and the changes are guaranteed to

work for the end-user. The cli is considered stable, and so tests written in

terms of it are unlikely to require updating as the internals of the software

change over time.

It may be impractical to sufficiently examine some changes this way. For

example, the number of cases to test and the running time of each test may be

too high. It may also be difficult to contrive circumstances to cover every

line of the change. If this is the case, next you can consider also writing

unit tests that work more directly on the changes.

It is important to write unit tests in such a way that they do not break due to

changes unrelated to what they are meant to test. For example, if the test

relies on a lot of BuildStream internals, a large refactoring will likely

require the test to be rewritten. Pure functions that only rely on the Python

Standard Library are excellent candidates for unit testing.

- - -

Would like to hear your thoughts on whether this is an accurate view of things as they are now, or if it falls close enough in a good direction.

Cheers!

Angelos

Follow-Ups:
- Re: [BuildStream] Guidance on unit tests
  - From: Tristan Van Berkom