Re: Looping GET to http://gnome.org, high load

[Owen Taylor <otaylor redhat com>]

On Sun, 2006-09-03 at 08:45 -0400, Owen Taylor wrote:
> [ Note - after making the changes below the load is still high - around
>   80 - after restarting httpd, so I'm investigating more ]

Don't have this figured out yet.

 * The high load seems to correlate with many processes in the D state
   (uninterruptable sleep, probably IO)
 
 * This number seems to periodically skyrocket up and down:

[root window sites.d]# ps ax | grep D | wc
      9     113    1096
[root window sites.d]# ps ax | grep D | wc
    151     910    8132
[root window sites.d]# ps ax | grep D | wc
     19     201    1944
   
   A few seconds apart.

 * At times we are getting messages like:

Sep  3 09:04:39 window kernel: nfs: server container-back not responding, still trying

   In /var/log/messages.  

   So, it seems it might be related to NFS problems

 * But we only have 10-20 anoncvs proceses running. So how can we get 150 processes 
   waiting on NFS, since really only /cvs/GNOME is NFS mounted?
   
   The majority of the other processes in the D state seem to be httpd processes.

 * Stracing a few httpd processes, and watching when they block unexpectedly, it
   seems to be almost always when writing to the httpd log files.  

   (Mostly the art.gnome.org log, but I think that's just because it's the most
   heavily hit site. I've seen it on other log files as well.)

   This is strange to me ... why would appending a few hundred bytes to a log
   file block? I don't think we're handling more than maybe 10 web requests / sec
   total.

Right now not clear to me what's going on or how to improve it.

                                             - Owen