Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

[Christian Hergert <chris dronelabs com>]

Hi,

What you mentioned is good information to start hunting.  Was the CPU 
time related to IO wait at all?  Always get accurate numbers before 
performance tuning.  "Measure, measure, measure" or so the mantra goes.

Unfortunately, the symptom you see regarding IO will very likely change 
under a different processing model.  If the problem is truly CPU bound 
then you will only be starting IO requests after you were done 
processing.  This means valuable time is wasted while waiting for the 
pages to be loaded into the buffers.  The code will just be blocking 
while this is going on.

What could be done easily is every time an item starts processing it 
could asynchronously begin loading the next image using gio.  This means 
the kernel can start paging that file into the vfs cache while you are 
processing the image.  This of course would still mean you are limited 
to a single processor doing the scaling.  But if the problem is in fact 
cpu bound, that next image will almost always be loaded by time you 
finish the scale meaning you've maximized the processing potential per core.

To support multi-core, like it sounds like you want, a queue could be 
used to store the upcoming work items.  A worker per core, for example, 
can get their next file from that queue.  FWIW, I wrote a library, 
iris[1], built specifically for doing work like this while efficiently 
using threads with minimum lock-contention.  It would allow for scaling 
up threads to the number of cores and back down when they are no longer 
needed.

Cheers,

[1] http://git.dronelabs.com/iris

-- Christian

Mark wrote:
> On Fri, Aug 28, 2009 at 10:45 PM, Christian Hergert<chris dronelabs com>  wrote:
> > Have you profiled to see if the bottleneck is CPU bound?  If its IO bound,
> > you will only cause more contention by adding threading.
> >
> > At minimum, using a thread (or async gio) to load files and another thread
> > that just thumbnails might be a good idea.
> >
> > Cheers,
> >
> > -- Christian
> >
> > Mark wrote:
> > >   is there anyone here that has experienced with this? is there
> > > anyone i an work with on this? actually anyone that could help me
> > > through this? i'm brand new in glib and new to programming in general.
> > > I would like to give this a try but i'm afraid i can't do this alone.
> > > So any information about this, help and thoughts would be nice.
>
> Hi,
>
> I haven't done io profiling but i did calculate the disc usage for
> those 1927 files. and every benchmark was WAY below what my hdd could
> handle (Spinpoint F1 1TB hdd and it can handle roughly 100MB/sec).
>
> As for the CPU. i did "profiling" there. i just opened up gnome's
> system monitor tool and just watch it during the benchmark. That gave
> the result of 1 core running at 100% and 3 other cores idle thus there
> is a cpu bottleneck with thumbnail generating.
>
> About your suggestion for loading in a thread and thumbnailing in
> another thread.. don't you get in big problems if the pc is
> thumbnailing faster then it's loading? Or is that a impossible
> scenario?
>
> Mark.