Re: [Tracker] The tracker-store and tracker-store-queue branches

[Jamie McCracken <jamie mccrack googlemail com>]

great stuff guys

thanks

jamie

On Thu, 2009-05-21 at 11:27 +0200, Philip Van Hoof wrote:
> Hi guys!
>
> Passionate observers of the project have probably noticed that JÃrg and
> me have started working on several new branches. 
>
> We started out with calling them architecture-something followed by
> branches called tracker-store-something. Right now we have tracker-store
> and tracker-store-queue.
>
> These branches are about a rearchitecture that we are working on.
>
> The rearchitecuture's main ideas are:
>
> 1) The daemon
>
> Transforming trackerd into tracker-store. This means that tracker-store
> wont have indexing nor monitoring code. It'll be your SPARQL update and
> query servant that knows about the Nepomuk ontologies.
>
> I like to refer to this as the separation of church and code: it removes
> all inotify from the SPARQL service and moves that to the indexer.
>
> This makes it for example more easy to replace the indexer, to have
> different kinds of metadata servicing processes. It also gives
> integrators the opportunity to have a SPARQL+Nepomuk store and query
> service without an indexer.
>
>
> 2) The indexer
>
> Transforming the indexer into a completely standalone entity. This means
> that the indexer will do the monitoring too. Right now the indexer
> doesn't do this file monitoring, trackerd does it.
>
> Among the ideas the guys want to split the indexer up into multiple
> components. Most of this work depends on finishing tracker-0.6 as it'll
> be mostly the same people who are now working on that branch who'll
> passionately do most of the work on the rearchitecture of the indexer.
>
> Their plans all sound very cool to me.
>
> 3) The extractor
>
> Making the communication between the big three (store, indexer and
> extractor) more efficient is also among the plans.
>
> Right now you can compare the communication with a bunch of roundtrips
> (in both directions) between all three processes. We are envisioning a
> better structure where the communication is more like a triangle.
>
>   1. The indexer will monitor and find work for the extractor to do
>   2. The extractor gets instructed to extract metadata and ...
>   3. It sends it to tracker-store, which stores it
>
> 4) Your project's metadata delivery component 
>
> The new architecture is also for all those crazy ideas floating around
> in the community: Zeitgeist, GeoClue, etc. They'll be telling us about
> metadata "news": as soon as they know more about something, they'll just
> us. And we'll store it as Nepomuk and make it accessible over SPARQL.
>
> - Zeitgeist is a good example of this: its backend implements all sorts
>   of funky algorithms for translating user behaviour into computable and
>   meaningful metadata information. Like history of document usage.
>   Linking up relevance based on what the user was focusing on while
>   working with a specific document. I don't know (it all sounds
>   crazy-cool to me, just do it guys!).
>
> - There's a GSoc for collecting your "online metadata". For example from
>   Facebook, your favorite Youtube movies, etc. Cooool! And the student
>   is a really clever guy I noticed. So this will eventually just work.
>   You'll see! Awesome idea, I like this very much.
>
> x) Development methodology
>
> We plan to do this rearchitecture step by step. Taking small steps at
> the time. This means that you'll see certain crazy decisions being made
> within branches. Choices that might not always work well.
>
> It's of course not the idea that we'll also leave it that way. The idea
> is of those that we want to focus on a certain aspect of the architec-
> ture before we replace something else that we have put in place quickly
> to enable ourselves focusing.
>
> We will work as much as possible using branches. Each such branch can be
> considered, for our contributors, as such an area of focus. We plan to
> increase communication about the plans we have soon: We're still in a
> bit of experimental phase, but the overall idea has been solidifying
> lately.
>
> I think that with tracker-0.6 being prepped, and being almost finished,
> that sooner or later there's going to be more and more focus on the
> tracker-store branches.
>
> Here's a blog item that I wrote about these rearch plans earlier:
>
> http://pvanhoof.be/blog/index.php/2009/04/24/tracker-our-near-future-plans
>
>
> We hope that our community contributors will somehow find a way to join
> us in our efforts. We're trying really hard to keep you guys involved.
> But being a team of people, we're also working pretty fast nowadays.
>
> Huge ideas combined with a great team of developers equals huge changes.
>
> Understanding communication is key, I'll try to continue increasing it.
>
>
> Good, now back to your keyboards and start hacking!
>
> Cheers