Re: [BuildStream] Dropping support for `setup.py test`

[Chandan Singh <chandan chandansingh net>]

Hi Tristan,

Thanks for getting back. Some comments inline.

On Fri, Dec 6, 2019 at 3:09 AM Tristan Van Berkom
<tristan vanberkom codethink co uk> wrote:
...
> The `setup.py test` invocation is important precisely *because* it does not install any specific set of 
> dependencies automatically or run in any virtual environment.

I don't think this is entirely correct. `setup.py test` will not
install any build requirements (like Cython) the way we have defined
it in pyproject.toml. But, it will absolutely install any and all of
BuildStream's dependencies and testing dependencies, if they aren't
already installed. Compared to tox, the only difference being that
they will not be installed in a virtual environment, but as eggs in
current directory.

> This is the correct testing path we need to have for the integrator workflow (not for the developer 
> workflow), as this allows the integrator to properly test how BuildStream will be have once it is 
> integrated with your system and installed on your machine.

Thanks for stating this. This is what I suspected but wasn't 100%
sure. I'd agree that this is a valid use-case. What I fail to
understand is how `setup.py test` is solving this. As mentioned above,
it will install any missing dependencies so there's no guarantee that
the integrator is testing how BuildStream will actually be installed.

If the use case is that the integrators are already expected to have
all the dependencies installed in some environment, then why do we
need this additional inderiction of `setup.py test`? Can't we just
call pytest at that point?

My understanding of `setup.py test` is that it's advantage was to make
the dependencies available for testing, without having them installed
globally. But if we are not leveraging that, we are only using it as
an indirection for invoking pytest.

> If `setup.py test` invocations are really going to be removed from setuptools, then certainly there must be 
> an alternative to replace it (tox without venv ?) because it really does continue to serve a real purpose, 
> not only for BuildStream.

As far as I can tell, the options are to either use things like tox,
or just expose the underlying test runners like pytest.

Thanks,
Chandan