Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

[Mark <markg85 gmail com>]

On Fri, Aug 28, 2009 at 11:38 PM, David Zeuthen<david fubar dk> wrote:
> On Fri, 2009-08-28 at 23:15 +0200, Mark wrote:
>> I haven't done io profiling but i did calculate the disc usage for
>> those 1927 files. and every benchmark was WAY below what my hdd could
>> handle (Spinpoint F1 1TB hdd and it can handle roughly 100MB/sec).
>
> Uhm, wait, that's only true for sequential reading. For random IO (which
> is true for thumbnailing) on rotational media, the throughput is much
> much lower (probably on the order 1MB/s) due to seek times. It's much
> better on SSD but not everyone has a SSD.

I know. but thanx for pointing it out.
>
> You also want to remember to drop all caches before profiling this - the
> relevant file here is /proc/sys/vm/drop_caches - see proc(5) for more
> details.
>
> It might be useful to sort the queue of files waiting to be thumbnailed
> according to the block position of the inode (see the readahead sources
> for how to do that). But since a file is not necessarily laid out in
> order on the disk (ext4 helps a bit here though) [1], there might be a
> lot of seeking involved anyway. So it might not help at all.
>
> (And IIRC, Nautilus already reorders the thumbnail list to give priority
> to files currently visible in the file manager. It would probably look
> weird if this happened out of order - so the visibility sort would win
> compared to the block position sort.)
>
>     David
>
> [1] : And this is most definitely not true for files downloaded via
> Bittorrent - which is a very normal use case for video files (which we
> thumbnail)
>
>>
>> As for the CPU. i did "profiling" there. i just opened up gnome's
>> system monitor tool and just watch it during the benchmark. That gave
>> the result of 1 core running at 100% and 3 other cores idle thus there
>> is a cpu bottleneck with thumbnail generating.
>>
>> About your suggestion for loading in a thread and thumbnailing in
>> another thread.. don't you get in big problems if the pc is
>> thumbnailing faster then it's loading? Or is that a impossible
>> scenario?
>>
>> Mark.
>> _______________________________________________
>> gtk-devel-list mailing list
>> gtk-devel-list gnome org
>> http://mail.gnome.org/mailman/listinfo/gtk-devel-list
>
>
>


On Fri, Aug 28, 2009 at 11:49 PM, Christian Hergert<chris dronelabs com> wrote:
> Hi,
>
> What you mentioned is good information to start hunting.  Was the CPU time
> related to IO wait at all?  Always get accurate numbers before performance
> tuning.  "Measure, measure, measure" or so the mantra goes.

Perhaps a stupid question but what is a good way of profiling io? cpu
is easy but i've never done io.
In this case my hdd is certainly able to do more then 10 thumbnails
per second however i could see a potential issue when someone with a
slower hdd and a faster cpu then mine is thumbnailing a lot of images.
There the hdd will likely be the bottleneck.
>
> Unfortunately, the symptom you see regarding IO will very likely change
> under a different processing model.  If the problem is truly CPU bound then
> you will only be starting IO requests after you were done processing.  This
> means valuable time is wasted while waiting for the pages to be loaded into
> the buffers.  The code will just be blocking while this is going on.

And how can i test that?
>
> What could be done easily is every time an item starts processing it could
> asynchronously begin loading the next image using gio.  This means the
> kernel can start paging that file into the vfs cache while you are
> processing the image.  This of course would still mean you are limited to a
> single processor doing the scaling.  But if the problem is in fact cpu
> bound, that next image will almost always be loaded by time you finish the
> scale meaning you've maximized the processing potential per core.

That sounds like a nice way to optimize it for one core. But could
there be any optimization possible in my case? since i have 100% cpu
usage for one core with just the benchmark.
>
> To support multi-core, like it sounds like you want, a queue could be used
> to store the upcoming work items.  A worker per core, for example, can get
> their next file from that queue.  FWIW, I wrote a library, iris[1], built
> specifically for doing work like this while efficiently using threads with
> minimum lock-contention.  It would allow for scaling up threads to the
> number of cores and back down when they are no longer needed.
>
That sounds very interesting.
Just one question about the queue. Would it be better to thread the
application (nautilus) or the library (glib)? If your answer is the
library then the queue has to be passed from nautilus to glib. I would
say glib because all application have benefit from it without
adjusting there code.
> Cheers,
>
> [1] http://git.dronelabs.com/iris
>
> -- Christian
>
> Mark wrote:
>>
>> On Fri, Aug 28, 2009 at 10:45 PM, Christian Hergert<chris dronelabs com>
>>  wrote:
>>>
>>> Have you profiled to see if the bottleneck is CPU bound?  If its IO
>>> bound,
>>> you will only cause more contention by adding threading.
>>>
>>> At minimum, using a thread (or async gio) to load files and another
>>> thread
>>> that just thumbnails might be a good idea.
>>>
>>> Cheers,
>>>
>>> -- Christian
>>>
>>> Mark wrote:
>>>>
>>>>  is there anyone here that has experienced with this? is there
>>>> anyone i an work with on this? actually anyone that could help me
>>>> through this? i'm brand new in glib and new to programming in general.
>>>> I would like to give this a try but i'm afraid i can't do this alone.
>>>> So any information about this, help and thoughts would be nice.
>>
>> Hi,
>>
>> I haven't done io profiling but i did calculate the disc usage for
>> those 1927 files. and every benchmark was WAY below what my hdd could
>> handle (Spinpoint F1 1TB hdd and it can handle roughly 100MB/sec).
>>
>> As for the CPU. i did "profiling" there. i just opened up gnome's
>> system monitor tool and just watch it during the benchmark. That gave
>> the result of 1 core running at 100% and 3 other cores idle thus there
>> is a cpu bottleneck with thumbnail generating.
>>
>> About your suggestion for loading in a thread and thumbnailing in
>> another thread.. don't you get in big problems if the pc is
>> thumbnailing faster then it's loading? Or is that a impossible
>> scenario?
>>
>> Mark.
>