RFC: collecting gnome-shell performance reports

[Owen Taylor <otaylor redhat com>]

I've been working recently on an automated performance testing framework
for GNOME Shell. You can see the GNOME Shell part of it at:

 https://bugzilla.gnome.org/show_bug.cgi?id=618189

The basic idea of an automated performance test:

 * You provide a Javascript script that drives the shell
 * While the script is running, timings and statistics are recorded
   to an event log.
 * You provide additional Javsascript code that computes "metrics"
   based upon replaying the event log.

Metrics includes things like "How much memory does the shell use after
startup". "What frame rate is observed when animating zooming out
to the activities overview".

At the end of the performance test, a report is written out to a file.

The next step after that is to upload it to a server where we can
compare across different systems and on individual systems over time.

I've begun some work on writing a server to collect and display these
reports with the idea to deploy that on gnome.org. It's not in a
functional state yet, but wanted to throw some questions/ideas out there
for discussion early:

 * What I've written so far is in Django. I picked that:

   - Because we're using that a bit on GNOME already (Mango rewrite)
   - Because it seemed simple and self-contained
   - Because compared to some over-engineered clever framework I
     might come up with myself, it seems likely someone else could
     pick it up and continue the maintenance.

   My hope that it would be easy to get something going seems to be
   working out. I'm not loving either the templating language or
   the ORM of Django ... I think I'd find them pretty limiting and
   annoying on a big project. But they should be OK for this.

 * I'm uncertain where it would live in the GNOME web hierarchy.
   Is it:

    http://perf.gnome.org/shell
     (Hope to add similar things for other parts of GNOME)

    http://shell.gnome.org/perf
     (Hope to add things like a Shell extension repository site)

    http://shell-perf.gnome.org
     (Just make it a standalone vhost)

    http://people.gnome.org/otaylor/shell-perf/

   One advantage of not using a separate vhost for individual 
   small apps like like this is being able to HTTPS-secure them
   without having to get separate certificates.

   (I'm not even sure if it's possible to have multiple HTTPS
   sites on the same IP - there used to be problems with HTTPS-based
   vhosts, but maybe that has been resolved?)

 * Which system does it live on? It obviously is conceptually similar
   to the build master function of fixed.gnome.org, but I don't think
   there is any synergy, and in general I'd rather not set up anything
   new on physical machines.

   Would this go on wiki.gnome.org as our "dynamic python hosting" VM?

 * Disk usage for this is medium. Currently the performance reports
   are 4k each, but they'll get bigger as we record more data.

   Rough estimate for an upper bound:

    100k/report * 
    10 systems * 
    1 report/day/system * 
    365 days/year * 
    2 years of data retained

   Or about 1GB.

   (The reason that the reports are big is that they include the event 
   log - not completely clear if that will be useful, but being able
   to chart memory usage over the course of a run would potentially
   be pretty interesting in figuring out why we suddenly started
   using more memory.)

Anyways, just wanted to give an initial heads up, and get any immediate
feedback people have. Once I get a little further, I'll put it up for
test purposes on my server so people can take a look.

- Owen