On Wed, 25 Oct 2006 12:40:27 -0500, Federico Mena Quintero wrote: > Cairo has a very nice testing framework. We can surely steal ideas from > it. I don't know that cairo's "make check" stuff has much worth stealing directly. But I will share some experiences at least. There are some things I really don't like in cairo's "make check" suite right now: 1. We are exporting a (very few, very tiny) test-specific functions in the library just so the test suite can get at it. These are explicitly not "part of" the API, but they do exist. They're mostly on the order of flipping an internal bit to change the behavior to exercise some specific piece of the library. But this approach really doesn't scale. It would probably be better to do this kind of thing in a "test build" as Havoc recently mentioned, (cairo currently doesn't have such a build). 2. The tests take forever to link. Each test right now is a separate program. I chose this originally so that I could easily execute individual tests, (something I still do regularly and still require). The problem is that with any change to the library, "make check" goes through the horrifically slow process of using libtool to re-link the hundred or so programs. One idea that's been floated to fix this is something like a single test program that links with the library, and then dlopens each test module (or something like this). Nothing like that has been implemented yet. We do have log files and HTML output, both of which are very useful, but the format of each is very ad-hoc and nothing that I would recommend anybody copying. That stuff is also very strongly oriented toward tests which can be verified by "visual diff" of two images---this works well for cairo, of course, but wouldn't generalize well. We also did some work on hooking gcov up to our makefiles and getting lovely HTML reports of how much coverage their is in the test suite. As Havoc points out, this is really important, (but sadly, we haven't yet made a good effort at using this data and writing directed tests to improve the coverage). I'm not even 100% sure the gcov stuff in the Makefiles works right now, but it might still be useful for someone looking to how to start that kind of thing, (though the gcov documentation might be even better). > > - for all widget types, create and destroy them in a loop to: > > a) measure basic object setup performance > > b) catch obvious leaks > > (these would be slowcheck/perf tests) > > Yeah. GtkWidgetProfiler (in gtk+/perf) will help with this. Manu > Cornet has improved it a lot for his theme engine torturer, so we should > use *that* version :) Something that is worth stealing is some of the work I've been doing in "make perf" in cairo. I've been putting a lot of effort into getting the most reliable numbers out of running performance tests, (and doing it is quickly as possible yet). I started with stuff that was in Manu's torturer with revamping from Benjamin Otte and I've further improved it from there. Some of the useful stuff is things such as using CPU performance counters for measuring "time", (which of course I didn't write, but just got from liboil---thanks David!), and then some basic statistical analysis---such as reporting the average and standard deviation over many short runs timed individually, rather than just timing many runs as a whole, (which gives the same information as the average, but without any indication of how stable the results are from one to the next). The statistical stuff could still be improved, (as I described in a recent post to performance-list), but I think it is a reasonable starting point. Oh, and my code also takes care to do things like ensuring that the X server has actually finished drawing what you asked it to, (I think GtkWidgetProfiler does that as well---but perhaps with a different approach). My stuff uses a single-pixel XGetImage just before starting or stopping the timer. > > as always, feedback is appreciated, especially objections/concerns > > regarding the ideas outlined ;) Excellent stuff. Testing is really important and often neglected. Never forget the truth that Keith Packard likes to share often: Untested code == Broken code -Carl
Attachment:
pgpafVpiAJYRj.pgp
Description: PGP signature