[Tracker] Moving crawling to the indexer

[Martyn Russell <martyn imendio com>]

Hi all,


PREFACE:
========

So I have been looking at one of the remaining issues we have before
releasing the next version of Tracker upstream. As Jamie outlined it it is:

  1) enumerating and crawling directories needs to be done in the
  indexer (and pass directories to watch back to the daemon). Daemon can
  then run is nice 0 and normal ionice instead of nice 19 as only cpu/io
  heavy ops will be searches and queries which need to be fast as
  possible


THE PROBLEM:
============

I have spent close to a week trying to do this and finding out how easy
it will be to do. And it isn't as easy as I first thought.

To summarise, the problem is essentially where all the status handling
is done (and avoiding duplication of code) and the inter-process
communications.

After discussing this with Carlos this morning, we agreed to put it to
the mailing list to get some feedback about the way forward on this one.


THE IDEA:
=========

My idea was that the daemon, which is _ALWAYS_ running, should hold all
state information and the indexer shouldn't since it is temporary, not
always running. By state information, I mean:

- Are we in READONLY mode
- Are we running?
- Are we indexing for the first time?
- Are we merging?
- Are we paused manually (by the user)?
- Are we paused for IO (due to Monitor events).
- Is the disk full?
- Is the battery low?
- What state are we in generally (i.e. INDEXING, IDLE, etc).
- How many items have we processed?
- How long have we been indexing?
- How long before we are finished (estimated)?

There could be a "State" DBus interface to query which would super seed
the current status APIs after the next release.

There would be 2 modules, one for all the state handling and monitoring
and one for the actual states the application can be in (INDEXING, IDLE,
etc). The later could possibly be shared in libtracker-common (see below
where I discuss this more).


HOW IT WORKS NOW:
=================

All of these things are pushed up to the user in some form or another.
Right now they are split over the indexer and the daemon and the indexer
currently propagates some of them up, others are kept in the daemon already.

The indexer itself ALSO needs a state machine if we move the processor
into it. So it can know if it is crawling, indexing, merging, idle, etc.


ADVANTAGES:
===========

- Not having to start the indexer to know that the disk is full and it
will just exit or sit there idle for 300 seconds.
- Not having to start the indexer to realise the battery is low and we
won't be indexing anyway.
- State will all be in one place. This should minimise the amount of IPC
traffic to keep up to date with the indexer's reasoning for running or
not running and its state.


DISADVANTAGES:
==============

- The indexer wouldn't have any checks in place if it ran stand alone,
it would depend on the daemon telling it to stop or pause if (say) the
disk space runs out or we are low on battery.
- The daemon would have to have to control when the indexer is
paused/stopped due to a few more states - that would need working on.


OTHER IDEAS:
============

There has been some thought that we could have a TrackerState in
libtracker-common. I am not sure it really fits to put this somewhere
centrally. There are some shared states the indexer and daemon would
have, but not all are necessary for both. Having the daemon's states in
the common library might be good for the applet or other applications to
know what the status ID means when they use a DBus API (or perhaps the
libtracker API).


CONCLUSION:
===========

The work to put the crawler in the indexer is not going to be as quick
as I had thought. I think it would take about 2 or 3 weeks for 2 people
(Carlos and I) to implement, test and be happy with. For that, we would
start a new branch.


COMMENTS:
=========

What are people's thoughts about how we maintain state here?

-- 
Regards,
Martyn