Re: [Evolution-hackers] Camel Manifesto

[Sankar P <psankar gnome org>]

On Sat, Nov 21, 2009 at 12:51 AM, Jeffrey Stedfast <fejj novell com> wrote:
> Matthew Barnes wrote:
>> With work on Bonobo removal wrapping up, I've finally started taking a
>> closer look at Camel (Evolution's mail storage and networking library)
>> and laying out plans for where I'd like it to go over the short and long
>> term, with the ultimate goal of splitting it off as a useful standalone
>> GNOME library (but we're a long way from that).
>>
>> As usual I'm taking a bottom-up approach, starting with basic cleanup
>> chores (both code and development policies) and building up from there.
>> Here's some of my thoughts:
>>
>>
>> Backward Compatibility Policy
>> -----------------------------
>>
>> A reverse dependency search in Debian and Fedora reveals the only
>> external projects currently linking to Camel are mail-notification,
>> evolution-jescs and Anjal (please correct me if I've missed any).
>>
>> That tells me that until Camel moves out of its parents' basement and
>> matures into an real, honest-to-goodness library, fixing its interface
>> is more important than maintaining backward compatibility.  Deprecation
>> periods for obsolete API are not necessary, in my opinion.  The few
>> external projects linking to Camel will just have to keep up with the
>> changes every six months.
>>
>
> I think it's a worthy goal to separate Camel out of Evolution.
>
>> That's not a license to go hog wild though.  Some caveats:
>>
>> 1) The soname -must- be kept accurate.  If you break the API or ABI,
>>    increment the soname when you commit the break.  It doesn't matter if
>>    the break doesn't affect external projects, nor even if it's already
>>    been incremented since the last point release.  Bump it anyway.
>>    "Always bump" is an easy policy to remember.  It makes our own daily
>>    development run smoother, and helps ensure a release doesn't slip out
>>    with an inaccurate soname.
>>
>>    If you're not sure if your patch requires a soname increment, please
>>    ask in IRC or Bugzilla.  Patch reviewers should try to remember too.
>>
>> 2) If you must break the API, try to do so in a way that things will
>>    fail noisily at build time rather than mysteriously at run time.  For
>>    example, if you want to change the behavior of an existing function,
>>    it's better to rename the function or change its parameter list so
>>    that stale Evolution code will fail to build.
>>
>> 3) Camel started life as a general purpose mail library and I'd like to
>>    try to get back to that.  Camel has become too Evolution-centric in
>>    my view, with too many quick-fix hacks for Evolution bugs that would
>>    not be appropriate for a general purpose mail library.  I will clean
>>    these up as I find them, but try to keep that in mind when altering
>>    the API yourself.
>>
>
> I'm not sure what Evolution-specific quick-fix hacks you mean?
>
>>
>> Migrate to GObject
>> ------------------
>>
>> Camel's homegrown type system will be replaced with GObject so that
>> introspection and D-Bus + language bindings are possible.  CamelObject
>> will remain (for now), but as a subclass of GObject.  The redundant
>> parts of CamelObject will be removed.
>>
>
> Worthy goals... there are some things like CamelObjectBag and such that
> don't have an equivalent in GObject.
>
>> I'm also following GLib and GTK+'s example of sealing up public instance
>> data in private sections and enforcing that only its top-level header
>> file be included outside of Camel (including the providers).  Unlike
>> GLib and GTK+, there will be no transition period.
>>
> Okay.
>
>> This will give us a lot more freedom to reorganize the library and
>> refactor code without disturbing the ABI.  Debugging is also easier when
>> you can trap data accesses through "get" and "set" functions.
>>
>
> Sure, but I don't think any code actually goes behind any get/set
> method's back at the moment (or at least didn't back when I worked on
> Camel). Regardless, I'm cool with the proposed changes...
>
>> I've been chipping away at this as a side project for the past year (it
>> was a good mindless activity when I got burned out on Bonobo removal),
>> and I recently published my results to git.gnome.org as a branch named
>> "camel-gobject".  The CamelObject conversion is finished -- including
>> all the boilerplate changes in the subclasses -- but I haven't finished
>> sealing up the API.
>>
>> The branch probably won't land until 2.31 at the earliest.  The backward
>> compatibility policies I described above would be in effect thereafter.
>>
>>
>> Kill CamelStream
>> ----------------
>>
>> This is a distant future goal and will have to happen gradually, but I
>> would like Camel to shift to a single-threaded design where all file and
>> network operations directly use or are derived from GIO's asynchronous
>> file and stream APIs.  SSL support is currently under development for
>> GIO, and that's the only missing piece I see at the platform layer.
>>
>> I realize this is a drastic course correction and will require rewriting
>> all the providers and much of the mailer code in Evolution, but I firmly
>> believe that the overuse of threads carries much of the blame for
>> Evolution's chronic instability over the years and that reversing that
>> trend first and foremost requires minimizing our use of threads for I/O
>> and relying more heavily on GLib's main loop.
>>
>>
>> Comments and constructive criticism encouraged.
>>
>
> I agree with Michael Meeks' concerns here. I also think there are much
> more important fish to fry which are also far easier to tackle.
>
> An IMAP rewrite (which supports IDLE) seems to be something a lot of
> users would be very thankful for, for example. The ancient IMAP code
> that is still being used by default (afaik) is probably one of the
> biggest sources of frustration for both users and maintainers (it also
> has a reputation for being the biggest source of race
> conditions/deadlocks in all of Evolution - which seems to be your
> primary reason for making Camel async ;-)
>

I absolutely agree. I am reminded of some of the code comments that
used to exist like: "There are so many broken IMAP servers out there",
"Hack to handle the broken Exchange 2005 IMAP access" etc.  Evolution
now has far more providers to support and far less people to work on,
than say two years ago.  So, I might actually suggest that we rely on
something like "offlineimap" to do the networking part. Evolution can
then just work on the maildir provided by the mail fetching daemon.
since offlineimap supports signals, you can map "get-mail" events
also. the only problem is that the service will be disconnected and
all mails will be fetched.

Though everyone likes to write new code, I am sure it will cause a lot
of maintenance problems with so many providers to support. Camel can
just be mail library that runs as a service and provides a single
point of api for other tools (Tracker/Beagle, OpenOffice, whatever).
Internally it makes use of offlineimap, smtp, GW-MUA for whatever mail
transport.

> The sqlite backend stuff could also use some work. As far as I'm aware,
> the tables are non-optimal. At one point I noticed that UIDs (the
> primary key in the table) were stored as strings - it would be better to
> store them as uint32s (yes, I know the gw and exchange backends do not
> use uint32s for UIDs) for performance reasons. It should be possible for
> these backends to have a second table which mapped the canonical uint32
> UID key to the key used by the servers.
>

Yes. when Srini and I started with the sqlite summary work, we planned
to implement the basic workflow first and then to improve the database
in more effective ways.  But sadly none of us work anymore on
Evolution directly and we weren't aware of these surprises when we
implemented the initial version of sqlite summary.

One important area that could improve immensely is Search. We now do a
"Like '<string>'" query on database often leading to slowed search
results. We can create more indices and once the user starts typing in
the text box, we can autocomplete adn do a EXACT Search in teh
database often leading to faster results. For instance, if I search
for Fejj now the query that  goes to db is LIKE  FEJJ. Whereas, if we
can autocomplete during the search and show fejj novell com, then in
the database it is faster to search with a EXACT match. this may not
look like a big improvement in theory but I believe it will be a big
usability improvement and can also of be use for other applications
that will rely on us, when camel becomes a service etc. Sadly, I was
never able to make the management understand the need for this when I
was in the team and also we never had the time to complete the
disk-summary migration and other related tasks etc.

> Additionally, the way Camel current works (even now, with the sqlite
> backend), when a folder is opened, the entire summary for the folder is
> loaded the same as it used to be before the sqlite changes. This seems
> rather... wasteful? Kinda defeats the purpose of using sqlite (or any
> other database backend). The main problem with the older format is that
> it did not allow random-access, which means we really needed to load the
> whole thing into memory for a number of reasons:
>
>    1. each record being a different size requires sequential access
> from disk
>    2. can't sort by sender, subject, date (or whatever) until you have
> the entire summary in memory
>    3. doing lookups on message UIDs would be prohibitively slow from disk
>
> With a db backend, none of these problems exist any longer. Some Camel
> APIs would likely need to change in order to support taking advantage of
> this new feature, but it's something that needs to be done.
>

IIRC, For showing in message list, we already fetch only whatever
records are needed and do not load the full summary onto the memory.
For instance, if you have selected "Unread mail" in the QuickShow
combo box, only unread mails' summary items are loaded.

>
> These things seem like much bigger wins for Camel's (and, by extension,
> Evolution's) usability/maintainability than making Camel async and are
> also far more trivial to implement.
>
> Just some things to think about...
>

I tend to agree with Fejj on this and I am big fan of threads. But I
no longer work on evo and knowing Matthew  I believe he knows what is
the best. All the best :-)

-- 
Sankar P
http://psankar.blogspot.com