[BuildStream] Update CI and testing process

[Chandan Singh <chandan chandansingh net>]

Hi,

In this thread, I wish to document some of the problems that developers face
due to the way our CI process currently works, and propose some changes to the
way we run our tests. If people agree, I'd be happy to make these changes but
I want to ensure that we are on the same page before I do so.

So, here it goes...

Current CI process
==================

Since it may not be obvious to everyone, the way things work currently is that
we have some testsuite Docker images that are provided by the docker-images
repo [1].  These images have all the dependencies that are required to build,
run and test BuildStream. These images get rebuild frequently, but we only
update their tags in BuildStream's `.gitlab-ci.yml` when we need to pick up
some changes.  This is most common when we change versions of any dependencies.
Due to the way it is setup, we essentially have to duplicate our dependencies
in the main BuildStream repo as well as the docker-images repo, which is
error-prone and not very intuitive.

> From what I understand, there are two reasons for doing this:

a. Since we don't need to install anything during the testing phase, it saves
   some time.

b. It ensures that we always run our tests with pre-determined versions of our
   dependencies.

Problems
========

Local reproduciblity
~~~~~~~~~~~~~~~~~~~~

It is not obvious which versions of dependencies the teststuite images are
running at a given tag. This makes it harder for anyone to debug CI failures
locally. It also makes it quite common for developers to hit the classic
problem of "works on my laptop but fails on CI".

Other than this, our `.gitlab-ci.yml` already is quite complicated in general,
which makes it hard for one to locally do "exactly whatever the CI is doing".

Updating versions of dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Since we have a setup.py in the main repo, folks are often tempted to change
that directly when restricting the version of any dependency. However, if the
testuite images don't already have the required version, the build will just
fail. This has often been a source of confusion in the past. A recent example
is this discussion on !909 [2].

The "right" way to do this today is to make the change in the docker-images
repo first, wait for it to be merged, find the new tag, and finally use the
new tag in your MR on the main BuildStream repo. This process is more complex
than it needs to be, but more importantly makes it harder to test new versions
of dependencies as they have to go via the docker-images repo.

Proposed solution
=================

There are a few things that I think we can do to reduce the responsibility of
the testuite images, and do everything in the main BuildStream repo to avoid
unnecessary complexity.

1. Do not install any python packages in the testsuite images. This will get
   rid of the hack (`pip install BuildStream && pip uninstall BuildStream`)
   that we currently use to install BuildStream's dependencies, and we won't
   need to maintain a duplicate copy of `dev-requirements.txt`.

2. Add `requirements.in` file to list our dependencies, that we currently do in
   `setup.py`.

3. Generate a `requirements.txt` file based on `requirements.in` that will have
   frozen versions of all dependencies.

4. Similarly, have `dev-requirements.in` and `dev-requirements.txt` for test
   dependencies.

5. Install `requirements.txt` and `dev-requirements.txt` before running any
   tests.

The `.in` files will provide the minimum version requirements, while the `.txt`
files will list exact versions of the packages that are known to work.

The benefit of having the requirements files is that it makes it very easy
for someone to locally test with the exact versions of dependencies that will
be used by CI. Additionally, it will get rid of the multi-step process to
update their versions. We can even provide a command for doing this in future
(like `make update-requirements`).

This will be somewhat slower than using an image with all dependencies
pre-installed, but not by a lot. Based on my experience, it should not take
more than a few seconds to install all the python dependencies.

Does this sound good to people, or does anyone has any objections?

---

The above proposal was to simplify the way in which we handle our
dependencies. I also have some other ideas around how we can simplify our
`.gitlab-ci.yml` so that it's running just a few commands (like `make tests`)
rather than how it looks right now (like manually packing and unpacking
tarballs). To do that, I was considering using `tox` [3] and `make` to simplify
it. So, wanted to ask if people would be open to introducing `tox` and/or
`make` in the mix?


[1]: https://gitlab.com/BuildStream/buildstream-docker-images
[2]: https://gitlab.com/BuildStream/buildstream/merge_requests/909#note_122431049
[3]: https://tox.readthedocs.io/en/latest/

Cheers,
Chandan