Re: [Tracker] tracker-indexer does not index all files

[Jamie McCracken <jamie mccrack googlemail com>]

> This is quite strange, Philip and me have been thinking a while and
> can't identify what could cause this. There are a few open questions:
>
> - Was it scanning a summary/mbox?
> - In that case, was is just opened for indexing, or did it happen in the
> middle of the processing
> - Does valgrind complain about something obvious?
>
> If you can reproduce it, I'd recommend removing all modules but
> evolution in $prefix/share/tracker/modules/ (mostly to save some time),
> delete everything in ~/.cache/tracker and run the indexer like:
>
> G_SLICE=always_malloc valgrind --leak-check=full ./tracker-indexer -p -v 3
>
> that should also save every debug message to
> ~/.local/share/tracker/tracker-indexer.log, it'd also be useful to see
> the last messages before the crash.


Another potential crasher - unlike trunk get_file_content does no utf-8
validation and also if file is bigger than MAX_TEXT cuts it off which is
likely to not land on a valid utf-8 word break

ideally do what trunk does and read file line by line so that we will
never have a partial utf-8 fragment and the resulting text can be
validated and converted from locale to utf-8 if necessary

ïThis needs to be fixed prior to merge!

jamie