Re: New module proposal: tracker

[Jamie McCracken <jamie mccrack googlemail com>]

The indexer part is optional

The main part tracker-store is just a database with querying and is to
be used by zeitgeist

If the consensus is that indexer is not suitable for inclusion then the
separate tracker-store should be considered for inclusion separately 

the store does not do any indexing or file monitoring nor does it cosume
significant resources 

jamie

On Tue, 2009-08-18 at 17:07 +0200, Lennart Poettering wrote:
> On Tue, 18.08.09 13:05, Martyn Russell (martyn lanedo com) wrote:
> 
> > Hi all,
> >
> > So we recently polled the tracker mailing list to make sure the core  
> > developers and others interested had an opinion on GNOME module  
> > inclusion for Tracker. You can see the thread here:
> >
> > http://mail.gnome.org/archives/tracker-list/2009-August/msg00007.html
> >
> > The response was positive. So I would like to propose Tracker as a new  
> > GNOME module.
> 
> Hmm. The beef I have with Tracker (and Beagle fwiw) is that they build
> something on infrastructure that currently is not good enough
> to sustain it: inotify. inotify is simply not suitable for recursively
> watching $HOME, but Tracker tries that nonetheless. And that is a big
> big failure, it should not do that.
> 
> There's something I like to call the "tracker paradox": if you have
> a large data set tracker is useless because inotify doesn't scale and
> the database is quickly out-of-date -- and if you have a small data
> set then you don't need a search engine and hence tracker is useless
> too.
> 
> As I see it the usefulness of Tracker stands or falls with the
> scalablity of inotify. As long as inotify does not natively support
> recursive watches tracker is not viable.
> 
> Right now installing tracker on a non-trivially sized $HOME has the
> effect of taking away all inotify watches and thus making inotify
> unavailable for other applications -- where they usally are more
> appropriately used, such as Nautilus. And all that even without fully
> working properly since if the limit of inotify handles is reached the
> view tracker has on the file system will necessarily become
> out-of-date quickly. Or more drastically spoken: installing Tracker on
> a machine with a non-trivially sized $HOME breaks Nautilus and other
> software.
> 
> And no, increasing the inotify limits is a band-aid, not a fix.
> 
> Oh and inotify is not the only area where the Linux file system layer
> is not ready to sustain Tracker, it's just the most obvious. Another
> thing that come to my mind is that we curently lack recursive mtimes
> so that tracker could identify changes on filesystems that happened
> while tracker was unable to access them (i.e. due to reboot, logout,
> hot unplug).
> 
> Really guys, before pushing forward with getting this into the desktop
> make sure that your infrastructure is actually suitable for what you
> want to do. And right now it simply is not!
> 
> All these things are fixable. Apple has shown that one can get all
> these things fixed. It's just a matter of someone sitting down and
> actually fixing the kernel. But as long as that hasn't happened you
> are roofing your house before you built its foundation.
> 
> As I see it Tracker is not ready for inclusion in the desktop. It
> might not be entirely Tracker's fault though, it's more the kernel's
> fault, but that doesn't change the conclusion.
> 
> Or in short: just f*ing fix the kernel first.
> 
> Lennart
>