Re: Common music database?

[Jamie McCracken <jamiemcc blueyonder co uk>]

Joe Shaw wrote:
> On Tue, 2006-04-18 at 14:58 +0100, Jamie McCracken wrote:
> > Tracker is significantly different from Beagle. Beagle AFAIK is just an 
> > indexer whereas Tracker is a combination Database/Indexer (the embedded 
> > mysql also being a combo database/indexer).
>
> This is the case right now, although the goal for Beagle has also been
> to act as metadata storage.
>
> These days, though, I'm not sure that's really the right way to go.
> Ideally you'd associate metadata with the file itself (probably via
> extended attributes) if you can.  

Thats obviously problematic  (permissions, NFS, EAs not present/enabled 
etc). But for me the killer to this is non-file objects like contacts as 
tracker allows all its services to have arbitrary metadata defined 
including keywords/tags against it so obviously that would not be 
possible with EAs.

The other issue is how you separate the useless junk from the 
interesting stuff (how will you know that a certain user defined 
metadata needs to be indexed?) and then there's the lack of datatypes of 
the metadata that would be present in EA's too.

For tracker, you can register a new metadata type for anything but we 
also need to know its datatype (if its indexable string, a non-index 
string, a numeric value or a datetime value) as they are stored/indexed 
differently.

This is probably less of an issue for Beagle as I assume you are 
hardcoding all your metadata (AFAIK Lucene's metadata is pre-set at 
design time and is not extensible at run time?). I guess this is why KDE 
and others use Sqlite for metadata whilst they use Lucene for full text 
searches.


External projects like Leaftag make me
> increasingly think that the indexer isn't the right place to be storing
> metadata, simply that it should pull it from external sources.

Well tracker is first and foremost a database and its goal is "one 
database to rule them all"!

I understand LeafTag will support multiple backends and I intend having 
Tracker as one of them.

The benefits of having one central DB are massive:

1) Faster and much more memory efficient - eliminates platform bloat.
2) Allows effortless cross querying of all data and metadata (via RDF Query)
3) Turns all the data into first class objects which are extensible with 
metadata (both derived and user defined)
4) Allows all objects to have keywords/tags applied to them
5) One consistent API to learn
6) Freedesktop. In theory it would allow you to change apps/desktops and 
still retain all your data

From http://live.gnome.org/ThreePointZero, Tracker fulfills some of the 
goals (as well as first class objects) like:

*Introduce Metadata throughout the desktop

Which basically means a full semantic desktop.

> That doesn't mean that I couldn't be convinced otherwise, of course. :)

Well that depends on whether you would want to become a metadata DB 
service (using sqlite and not lucene for the metadata) or alternatively 
if you dont want to reinvent the wheel, integrate with Tracker (you 
would need to fix the mono dbus bindings first though!).

One possibility is to have a gconf key in tracker that would pass all 
full text search queries to libBeagle and disable all watching/indexing 
being done by tracker (IE tracker would behave like a DB and not an 
indexer). On the other end, Beagle would need to populate Tracker with 
metadata whenever files changed (you would fire off async dbus calls so 
it would not slow down beagle's indexing).

I would like to add Tracker to gnome at some point and avoiding a turf 
war with beagle would be best for everyone (especially as Tracker offers 
much more than indexing). It means we can get search, indexing and 
powerful metadata capabilities into the core desktop and hopefully 
change everything for the better (like having hierarchiless file dialogs 
where apps simply state what mime types they want and tracker shows all 
matching files - I really hate all the dumb manual navigation thats 
needed at present!)


--
Mr Jamie McCracken
http://jamiemcc.livejournal.com/