Re: Ottawa - kernel hackers ...

[Michael Meeks <michael ximian com>]

Hi Alan,

On Fri, 2003-06-27 at 12:12, Alan Cox wrote:
> Fancy elevator algorithms and the readahead stuff people are fiddling with
> are as relevant to the desktop as big servers.

	Right; elevation is great if one can predict what is going to be read
next; unfortunately the kernel cannot be particularly well informed
about that AFAICS, it can only try to read bigger chunks.

> Soeren Sandmann wrote:
> > in the kernel is a way for applications to monitor pagefaults. This
> > would allow applications to generate a list of disk accesses during
> > startup. Then the next time they start, instead of waiting for what
> > is very likely going to be the same set of pagefaults, they just ask
> > for that data to be read in immediately.
>
> That actually doesnt help much. What you have to do is figure out the
> stuff used together and arrange the code to be clustered together
> better. At that point loading big chunks on page faults helps a lot.

	I'm curious as to why it doesn't help much. I'd have thought that
pre-loading a large number of pages which are known to be needed ahead
of time should be a sane strategy. Anecdotal reports suggest that on
some Win32 systems libraries are fully loaded before linking, and that
this makes life somewhat faster - all else being equal.

	It seems that the kernel has to be given more information to do a
better job, and simply trying to cram everything into big slurps in
user-land, and hoping the kernel will slurp more doesn't seem (to me) to
be likely to work as well as if it could accurately guess what files
would be read next by being told what's likely to happen by an informed
user-land.

	eg. (and I pick on GConf again) if you start up Gnome - it does in the
worst case in the order of a:

	time find ~/.gconf -name '*.xml' -exec cat {} > /dev/null \;

	Which on my (warm) machine is ~8 seconds, to read 180kb of data, from
cold it's substantially worse. It seems to me that some holistic idea of
the scope of what the machine is being asked to do could be used at
login to very substantially improve login / app start time.

> > [1] - it's quite amazing to quit X after a long session and see the X
> > background pixmap re-render from swap: a huge area of flat memory - with
> > uniform non-use for the last some hours - and the pixels are scattered
> > all over the disk - hmm, at ~1 block per line it takes the ~10 seconds
> > (1024*10ms) you'd expect to render. Just killall -9 nautilus to see.
>
> And once you have the data scattered it gets really fun because disk seek
> time hasnt really improved in years, so its 50Mb/sec linear, 2Mb/sec seeking
> hard.

	I appreciate that. What concerns me is that by using mental arithmetic,
performed while watch it re-render, you can see that the kernel is doing
~1 seek per 4kb block reading the single chunk back in. 

	It's certainly true that I understand nothing of the intricacies of
page ageing / swap management etc. however it would seem to me that
scattering a single, huge, uniformly unused chunk of memory across the
disk in 1000 tiny fragments points to some underlying brokenness
somewhere. [ RH 8.0 stock kernel ]. Of course - OTOH, perhaps the X
server is splitting it into tiles internally - but even so ...

> X or gnome could choose to keep file mappings of these objects in theory
> but currently don't

	I don't understand that; what would that entail. 

	Finally I suppose I'm most interested in tools to measure the problem;
can one measure the time the kernel is idling waiting for the disk heads
to move around easily ? presumably in my 'time' results from the above
gconf find:

real    0m8.736s
user    0m0.209s
sys     0m0.406s

	we can infer that it's the unaccounted for 8seconds that are seek time
on an unloaded system - but can those numbers be got more easily /
reliably ?

	Also - is there an easy way to totally empty any disk caching to easily
simulate cold start ( I believe an unmount / remount of a partition
could do that - but not wanting to re-partition makes that a painful
option ).

	Anyhow - thanks muchly for the assistance,

	Regards,

		Michael.

-- 
 michael ximian com  <><, Pseudo Engineer, itinerant idiot