Re: reftests

[Benjamin Otte <otte gnome org>]

On Thu, May 5, 2011 at 10:18 AM, Kristian Rietveld <kris gtk org> wrote:
> As we've already discussed on IRC some time ago, I would really like to see all
> GTK+ unit tests in one single place, instead of in several different places in the
> source code.  We really need people to run the unit tests more often and thus
> this needs to be made easy (like you also mention in your enumeration above),
> I don't think putting different unit tests at different places makes this easier.
>
I do agree with putting all things in one location. However, I do not
agree with "people need to run tests more often". I think running
tests is a job for machines.

> We can always distribute the unit tests as a separate tarball if that will help, can't we?
>
Well yeah, but then it still requires compilation of something. The
approach with "Here, open this in glade" and "Does it look like this
screenshot?" is a lot better, because it's very easy. I guess having
tests written in Python would come closest to this.

> Another question: why was gtk-reftest put in gtk+/tests/reftests/gtk-reftest instead of in gtk+/gtk/tests/gtk-reftest, with a subdirectory reftests containing the glade files?  Then on make check for the GTK+ unit tests, the reftests would automatically be executed as well.  Currently, you also need to compile a GTK+ checkout to use reftests, right?
>
I never liked the idea of putting tests in the same directory tree as
the actual source code, because in the good case it creates spam to
stdout (about entering directories I don't care about) and in the
worst case it actually compiles tests, so it not only spams stdout but
it also takes lots of time relinking. If I am actively working on
refactoring code I do not want to be annoyed by random test cases.
Currently this is still bearable, but when there's 10.000 tests that
each get relinked and that takes 5 minutes after every 1 line change
in gtknotebook.c or so, I am going to be really annoyed.
So I do think that tests, just like documentation should live outside
of the normal source tree. Which is why I put it there.

> These sounds like numbers I would expect.  What in GTest would need improvement to realize this?
>
GTest mainly needs usability improvements such as those you pointed
out by those error messages. (I'm sorry if I offended you by taking a
test written by you as the bad example, I just took a random file from
gtk/tests as an example of why our current approach is bad.) The
problem is that currently running tests (or a single test) manually is
complicated and oftentimes ends up with unparsable error messages that
often are no help in actually figuring out what got broken.
Also, invocation of test runners should be a lot simpler (see below).

> About organization, I think for one all GTK+ unit tests should be in one place (and the GDK tests in another place).
>
Yes, I would very much suggest tests/ for this. I don't care a whole
lot if we have tests/gdk and tests/gtk, because I think it's hard to
test GDK without also testing GTK as GTK has all the niceties that you
want to have when writing tests (like ui files) and it's very
cumbersome to write to the GDK API.

> Secondly, we also need to develop a consistent naming scheme for tests.  Unit tests currently have different ways of naming tests:
>
> /FilterModel/self/verify-test-suite:
> /expander/click-expander:
> /recent-manager/get-default:
> /tests/column-new:                       (these are for icon view)
> /Builder/Window:
>
I would attribute that to GTester. The path naming idea is kind of
useless, because what you actually need to remember is the name of the
test binary (so that you can run it), not the name of the actual test.
There is no way to run "all GtkLabel tests" by saying gtester -p
/tests/gtk/widgets/label and that way making it grab all tests in the
testsuite and make it run those that match this path. Instead you run
tests/gtk/widgets/label and that's it.
You can also see that gtk-reftest basically ignores the test paths -
or better: abuses them to store filenames in them so that you actually
get useful output when running a test, because it tells you the actual
file that failed.
So for this case we'd actually first need to have a use for these test paths.

Benjamin