Re: [Evolution-hackers] Introduction and Questions

[Jeffrey Stedfast <fejj novell com>]

On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote:
> On Thu, 2007-06-07 at 08:25 +0200, Gilles Dartiguelongue wrote:
> > Le jeudi 07 juin 2007 �1:53 +0300, Philip Van Hoof a �it :
> > > Without immediately writing to disk work, the download by itself will
> > > consume around 120 MB of RAM, and will most likely fail due to network
> > > timeouts and other such problems (it'll take a while, since Evolution
> > > fetches a ridiculous large amount of headers for each message for
> > > building its summary).
> > > 
> > isn't the imap-features plugin's goal to reduce/customize the amount of
> > downloaded headers ? Or is it that it's still not enough ?
> 
> It improves the situation by setting your url-string to have the
> "basic_headers" option. In the imap code of Camel, it will then ask for
> less headers (but still way too much).

if you set the option to "basic headers", it will only fetch the minimal
set of headers required to show the message-list:

"DATE FROM TO CC SUBJECT REFERENCES IN-REPLY-TO MESSAGE-ID MIME-VERSION
CONTENT-TYPE "

(actually, why is MIME-VERSION in there? it's useless)

> 
> A major improvement it certainly isn't.

it's not possible to do better w/o dropping features like message
threading.

In fact, the above minimalizing of header fetching already breaks the
quick context-menu "vfolder on mailing-list" and "filter on
mailing-list" features as well as making vfoldering on mailing-list
oodles slower (if it even still works after disabling the headers)

> 
> The best way is to ask for the ENVELOPE and the remaining info using the
> normal BODY.PEEK method.

Have you actually ever tested this theory? or did you just pull this out
of your proverbial?

You should note that Thunderbird does not use ENVELOPE, probably because
they found it was LESS efficient to fetch the ENVELOPE +
BODY.PEEK[HEADER.FIELDS (REFERENCES CONTENT-TYPE)] on many existing IMAP
servers in the wild.

Having myself tested this in the past, I don't recall it making much
difference one way or the other.

Sadly using ENVELOPE breaks on at least one proprietary IMAP server
which occasionally likes to send back responses with NIL as each and
every item, at least until a client has issued a BODY.PEEK fetch
request.

>  It should be possible to specify per folder
> which of the headers are to be fetched, to make it really efficient.

how do you propose calculating the list of headers each folder should
request? which headers would you eliminate conditionally?

> 
> The imap4 implementation seems to have an ENVELOPE parser, so I guess
> either it does it already or it will use ENVELOPE in future.

it fetches:

ENVELOPE BODY.PEEK[HEADER.FIELDS (Content-Type References In-Reply-To)]

and optionally the mailing-list headers

I believe it fetches IN-REPLY-TO (which is included in ENVELOPE)
explicitly because some IMAP servers tried to be smart and parse the
IN-REPLY-TO header for us and didn't handle embedded phrases which
certain broken free-software mailers like to send.

> 
> For a mobile device, that works over GPRS, you for example are usually
> not interested (not at all) in the mailinglist specific headers.

this is mobile-device specific tho, when I was implementing imap4 and
people started trying to use it, it was their #1 complaint (which is why
imap4 conditionally requests the mlist headers... tho that flag may be
hard-coded to TRUE now, I'm not sure)

> 
> It would also be possible to do it in multiple passes. For example get a
> modest list of headers, and after that get more and more.

but then you introduce a lot more complexity (e.g. scheduling such a
fetch)

slowness for the user (e.g. if he expects to be able to right-click and
select "vfolder on mailing-list", he'd have to wait if the info had not
yet been fetched... or even just selecting a mailing-list vfolder)

keep in mind also that this is a one-time fetch, so while opening a
folder for the first time might be slow (if you have 1000's of
messages), future folder selections would take a much more reasonable
amount of time to open (especially with CONDSTORE support which reduces
the FLAGS sync requirements, which is where the real slowness is when
opening a folder for most users)

> 
> In any case, none of the current Evolution code implements consuming the
> CONDSTORE capabilities of some modern IMAP servers (like MBox and
> Cyrus).
> 
> CONDSTORE is really going to make an enormous difference in bandwidth
> consumption with large folders. That's because you only need to fetch
> the flags of the changed messages to synchronise flags.

yes, this would be nice to have... but as far as I'm aware, it is still
only a draft standard (e.g. it could change).

I would probably not be opposed to a CONDSTORE implementation making it
into upstream camel, tho.

> 
> Note that Tinymail's camel-lite implements all the needed solutions to
> this bandwidth consumption problem. And that its code is, although a lot
> of work because the Evolution maintainers didn't seem interested at the
> time it was happening, definitely portable to upstream.
> 

some of your changes are broken wrt working with all imap servers in the
wild.

> Its implementation include solutions for the headers, it immediately
> saves the headers to disk and unlike Evolution's Camel it can resume
> partial summary downloads, it wont peek memory allocation during
> downloading, it implements Push E-mail using IMAP IDLE and it implements
> CONDSTORE.
> 
> Although I'm definitely not satisfied with any of either Camel nor
> camel-lite's IMAP code. The thing is that I'd much prefer a client-side
> implementation that does proper pipelining. For example Infotrope,
> Telomer and Polymer does this on an experimental basis (Infotrope is the
> library).
> 
> ps. In my opinion is also the imap4 project getting the majority of
> its
> design wrong.

thank you, oh king of software design.

Jeff