Re: New module proposal: tracker

[Ruben Vermeersch <ruben savanne be>]

On do, 2009-10-29 at 17:21 +0000, Martyn Russell wrote:
> > Working with 0.6 I've known nothing but pain (leaving aside
> > all the political pressure I've been facing for making my app so
> > dependent on Tracker) so please understand that I need some time to be
> > sure that 0.7 is completely different in that regard. Until then, I am
> > skeptical about Tracker to become integral part of GNOME.
> 
> I hear the same thing over and over again from people that used 0.6. 
> Carlos and I have spent 2 years refactoring and redesigning the *ENTIRE* 
> code base. Not only that Jürg and Philip have also been doing the same 
> since they came on the project (Philip in April 2008 and Jürg in 
> November 2008). Helping in this have also been Ivan Frade and Mikael 
> Ottela (at Nokia) since the project was taken on for the Maemo 5 platform.
> 
> During the time that Carlos, Mikael, Ivan and I have been fixing 0.6.x 
> issues for the Maemo platform, Philip and Jürg started working on 0.7. 
> which drastically changed the design and communication between all major 
> parts of Tracker. This came after a lot of review and discussion by the 
> core team.
> 
> Comparing 0.6. to 0.7. is folly, they are so different and any project 
> would be after 2 years of full time coding from 6 people and public 
> contributions on top of that.

>From a devil's advocate point of view: this might just mean that it got
worse (though I doubt it). Zeeshan raises concerns and while you go to
great lengths to explain that it is different, there is nothing that
might convince him that it is actually *better*.

> >    Also I share the concern of Lennart about inotify limitation and
> > from my talks with Tracker developers so far, it seems they
> > underestimate the importance of fixing this limitation wrt Tracker.
> 
> I agree it needs fixing, but there are a number of things to consider here:
> 
>   - FANotify is being worked on by Red Hat and will be in the kernel for
>     us to use at some point - and we will adopt it then (I believe it
>     almost made it into the latest Fedora but didn't so should be in the
>     next release)
> 
>   - We changed the locations that are indexed by default from $HOME to
>     use XDG user dirs for documents, desktop, music, pictures and videos.
>     So the focus has changed slightly to the things you most likely want
>     indexed instead of EVERYTHING. Of course adding EVERYTHING into the
>     config doesn't escape the fact that inotify is limiting us.
> 
>   - In the grand scheme of things, I don't think it is that important to
>     fix as Lennart says. I don't consider myself a normal user and I
>     don't even breach the inotify limit (I come close though with all the
>     project sources monitored). I personally would much rather have an
>     Tracker which is fast to query/update and has good coverage on the
>     metadata it extracts as a priority over supporting EXTREME use cases.
> 
> I do want this fixed but it has not been our focus because we have had 
> so many other issues to contend with first which were much more important.

Erm, wouldn't the lack of a recursive notification mechanism imply that
you have to do a directory crawl at startup to place notify watches on
each and every file?

That *is* an important issue, much more than the potential of running
into the limits in, as you said, extreme cases. Hugely occupying the
hard disk when the user wants to start using his/her computer is bad.
And please don't say "low priority I/O", disk seeks take time, so even
when the priority is low, other apps will be affected by it.

So the question is: does the indexer do a full directory crawl when
starting? Because in that case, I don't want it.

So not having recursive inotify (or something alike) can be a problem
for the indexer. Which does not say much about the data store. I think
it might be wise to consider the tracker data store and the tracker
indexer as separate components to propose, given the different issues
involved.

   R

-- 
Ruben Vermeersch (rubenv)
http://www.savanne.be/