[Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch
- From: Philip Van Hoof <spam pvanhoof be>
- To: evolution-hackers gnome org
- Subject: [Evolution-hackers] PATCH: Storing and fetching bodystructure, the patch
- Date: Tue, 14 Jul 2009 12:57:07 +0200
Hi there,
I would like Evolution to store IMAP's bodystructure as often as
possible. The reason for this is storing RDF graphs. Let me explain.
I wrote a EPlugin for Evolution that is compiled and distributed by the
Tracker project. This plugin fetches as much metadata about E-mails as
possible and pushes it into Tracker's RDF store.
Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
SPARQL as query language, and SPARQL Update as storage language.
We have enhanced the Nepomuk message ontology so that it can store the
entire structure of an E-mail. This means the structure that you can
request using the BODY or BODYSTRUCTURE FETCH requests. These return you
the MIME structure of the message in a pre-parsed skiplist format.
Using this info our plugin can reconstruct the message's skeleton, but
then in RDF as a graph.
I don't just talk. I wrote the patch to do this too. It's attached.
I tried to keep the changes small, and I implemented the migration code
so that the tables' schemas will automatically be converted. The patch
needs a good review, though. And testing.
This is the point of the patch (a stored bodystructure string):
sqlite> select bodystructure from 'INBOX/100' LIMIT 2;
("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "8BIT" 2304 53 NIL NIL NIL NIL)
("TEXT" "PLAIN" NIL NIL NIL "7BIT" 3829 80 NIL NIL NIL NIL)
sqlite>
Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
servers usually cache the body-structures. It wont slowdown an IMAP
server, one that is worth being called "an IMAP server", much.
It will add some bandwidth. But apparently isn't Evolution trying to
save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead
of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).
Here are some pointers about things I just wrote:
The Nemomuk message ontology
http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology
How we want this stuff in Tracker, how it could be used
http://live.gnome.org/Tracker/Documentation/EmailSparql
The Evolution plugin:
http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c
Let me know!
--
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
http://pvanhoof.be/blog
http://codeminded.be
diff --git a/camel/camel-db.c b/camel/camel-db.c
index be63c5b..504eea4 100644
--- a/camel/camel-db.c
+++ b/camel/camel-db.c
@@ -1138,7 +1138,7 @@ camel_db_create_message_info_table (CamelDB *cdb, const gchar *folder_name, Came
gchar *table_creation_query, *safe_index;
/* README: It is possible to compress all system flags into a single column and use just as userflags but that makes querying for other applications difficult an d bloats the parsing code. Instead, it is better to bloat the tables. Sqlite should have some optimizations for sparse columns etc. */
- table_creation_query = sqlite3_mprintf ("CREATE TABLE IF NOT EXISTS %Q ( uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , dirty INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT )", folder_name);
+ table_creation_query = sqlite3_mprintf ("CREATE TABLE IF NOT EXISTS %Q ( uid TEXT PRIMARY KEY , flags INTEGER , msg_type INTEGER , read INTEGER , deleted INTEGER , replied INTEGER , important INTEGER , junk INTEGER , attachment INTEGER , dirty INTEGER , size INTEGER , dsent NUMERIC , dreceived NUMERIC , subject TEXT , mail_from TEXT , mail_to TEXT , mail_cc TEXT , mlist TEXT , followup_flag TEXT , followup_completed_on TEXT , followup_due_by TEXT , part TEXT , labels TEXT , usertags TEXT , cinfo TEXT , bdata TEXT, created TEXT, modified TEXT, bodystructure TEXT )", folder_name);
ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
sqlite3_free (table_creation_query);
@@ -1193,7 +1193,12 @@ camel_db_migrate_folder_prepare (CamelDB *cdb, const gchar *folder_name, gint ve
/* Migration stage one: storing the old data */
- if (version < 1) {
+ if (version < 2) {
+
+ /* Between version 1-2 the following things are changed
+ * ADDED: bodystructure: text
+ * */
+
/* Between version 0-1 the following things are changed
* ADDED: created: time
* ADDED: modified: time
@@ -1208,7 +1213,7 @@ camel_db_migrate_folder_prepare (CamelDB *cdb, const gchar *folder_name, gint ve
ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
sqlite3_free (table_creation_query);
- table_creation_query = sqlite3_mprintf ("INSERT INTO 'mem.%q' SELECT uid , flags , msg_type , read , deleted , replied , important , junk , attachment , msg_security , size , dsent , dreceived , subject , mail_from , mail_to , mail_cc , mlist , followup_flag , followup_completed_on , followup_due_by , part , labels , usertags , cinfo , bdata , strftime(\"%%s\", 'now'), strftime(\"%%s\", 'now') FROM %Q", folder_name, folder_name);
+ table_creation_query = sqlite3_mprintf ("INSERT INTO 'mem.%q' SELECT uid , flags , msg_type , read , deleted , replied , important , junk , attachment , msg_security , size , dsent , dreceived , subject , mail_from , mail_to , mail_cc , mlist , followup_flag , followup_completed_on , followup_due_by , part , labels , usertags , cinfo , bdata , strftime(\"%%s\", 'now'), strftime(\"%%s\", 'now'), '' FROM %Q", folder_name, folder_name);
ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
sqlite3_free (table_creation_query);
@@ -1234,7 +1239,7 @@ camel_db_migrate_folder_recreate (CamelDB *cdb, const gchar *folder_name, gint v
/* Migration stage two: writing back the old data */
if (version < 1) {
- table_creation_query = sqlite3_mprintf ("INSERT INTO %Q SELECT uid , flags , msg_type , read , deleted , replied , important , junk , attachment , dirty , size , dsent , dreceived , subject , mail_from , mail_to , mail_cc , mlist , followup_flag , followup_completed_on , followup_due_by , part , labels , usertags , cinfo , bdata, created, modified FROM 'mem.%q'", folder_name, folder_name);
+ table_creation_query = sqlite3_mprintf ("INSERT INTO %Q SELECT uid , flags , msg_type , read , deleted , replied , important , junk , attachment , dirty , size , dsent , dreceived , subject , mail_from , mail_to , mail_cc , mlist , followup_flag , followup_completed_on , followup_due_by , part , labels , usertags , cinfo , bdata, created, modified, bodystructure FROM 'mem.%q'", folder_name, folder_name);
ret = camel_db_add_to_transaction (cdb, table_creation_query, ex);
sqlite3_free (table_creation_query);
@@ -1258,9 +1263,9 @@ camel_db_write_folder_version (CamelDB *cdb, const gchar *folder_name, gint old_
version_creation_query = sqlite3_mprintf ("CREATE TABLE IF NOT EXISTS '%q_version' ( version TEXT )", folder_name);
if (old_version == -1)
- version_insert_query = sqlite3_mprintf ("INSERT INTO '%q_version' VALUES ('1')", folder_name);
+ version_insert_query = sqlite3_mprintf ("INSERT INTO '%q_version' VALUES ('2')", folder_name);
else
- version_insert_query = sqlite3_mprintf ("UPDATE '%q_version' SET version='1'", folder_name);
+ version_insert_query = sqlite3_mprintf ("UPDATE '%q_version' SET version='2'", folder_name);
ret = camel_db_add_to_transaction (cdb, version_creation_query, ex);
ret = camel_db_add_to_transaction (cdb, version_insert_query, ex);
@@ -1340,7 +1345,7 @@ write_mir (CamelDB *cdb, const gchar *folder_name, CamelMIRecord *record, CamelE
/* NB: UGLIEST Hack. We can't modify the schema now. We are using dirty (an unsed one to notify of FLAGGED/Dirty infos */
- ins_query = sqlite3_mprintf ("INSERT OR REPLACE INTO %Q VALUES (%Q, %d, %d, %d, %d, %d, %d, %d, %d, %d, %d, %lld, %lld, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, strftime(\"%%s\", 'now'), strftime(\"%%s\", 'now') )",
+ ins_query = sqlite3_mprintf ("INSERT OR REPLACE INTO %Q VALUES (%Q, %d, %d, %d, %d, %d, %d, %d, %d, %d, %d, %lld, %lld, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, %Q, strftime(\"%%s\", 'now'), strftime(\"%%s\", 'now'), %Q )",
folder_name, record->uid, record->flags,
record->msg_type, record->read, record->deleted, record->replied,
record->important, record->junk, record->attachment, record->dirty,
@@ -1349,7 +1354,7 @@ write_mir (CamelDB *cdb, const gchar *folder_name, CamelMIRecord *record, CamelE
record->cc, record->mlist, record->followup_flag,
record->followup_completed_on, record->followup_due_by,
record->part, record->labels, record->usertags,
- record->cinfo, record->bdata);
+ record->cinfo, record->bdata, record->bodystructure);
/* if (delete_old_record)
del_query = sqlite3_mprintf ("DELETE FROM %Q WHERE uid = %Q", folder_name, record->uid); */
@@ -1757,6 +1762,7 @@ camel_db_camel_mir_free (CamelMIRecord *record)
g_free (record->usertags);
g_free (record->cinfo);
g_free (record->bdata);
+ g_free (record->bodystructure);
g_free (record);
}
diff --git a/camel/camel-db.h b/camel/camel-db.h
index 2df1616..ff5a33a 100644
--- a/camel/camel-db.h
+++ b/camel/camel-db.h
@@ -84,6 +84,7 @@ typedef struct _CamelMIRecord {
gchar *usertags;
gchar *cinfo;
gchar *bdata;
+ gchar *bodystructure;
} CamelMIRecord;
typedef struct _CamelFIRecord {
diff --git a/camel/camel-digest-folder.c b/camel/camel-digest-folder.c
index 94f8f22..6886312 100644
--- a/camel/camel-digest-folder.c
+++ b/camel/camel-digest-folder.c
@@ -196,7 +196,7 @@ digest_add_multipart (CamelFolder *folder, CamelMultipart *multipart, const gcha
continue;
}
- info = camel_folder_summary_info_new_from_message (folder->summary, CAMEL_MIME_MESSAGE (wrapper));
+ info = camel_folder_summary_info_new_from_message (folder->summary, CAMEL_MIME_MESSAGE (wrapper), NULL);
camel_pstring_free(info->uid);
tmp = g_strdup_printf ("%s%d", preuid, i);
info->uid = camel_pstring_strdup (tmp);
diff --git a/camel/camel-folder-summary.c b/camel/camel-folder-summary.c
index 5672b69..03e5384 100644
--- a/camel/camel-folder-summary.c
+++ b/camel/camel-folder-summary.c
@@ -116,7 +116,7 @@ static gint summary_meta_header_save(CamelFolderSummary *, FILE *);
static CamelMessageInfo * message_info_new_from_header(CamelFolderSummary *, struct _camel_header_raw *);
static CamelMessageInfo * message_info_new_from_parser(CamelFolderSummary *, CamelMimeParser *);
-static CamelMessageInfo * message_info_new_from_message(CamelFolderSummary *s, CamelMimeMessage *msg);
+static CamelMessageInfo * message_info_new_from_message(CamelFolderSummary *s, CamelMimeMessage *msg, const gchar *bodystructure);
static CamelMessageInfo * message_info_load(CamelFolderSummary *, FILE *);
static gint message_info_save(CamelFolderSummary *, FILE *, CamelMessageInfo *);
static gint meta_message_info_save(CamelFolderSummary *s, FILE *out_meta, FILE *out, CamelMessageInfo *info);
@@ -1165,6 +1165,9 @@ mir_from_cols (CamelMIRecord *mir, CamelFolderSummary *s, gint ncol, gchar ** co
mir->cinfo = g_strdup(cols [i]);
else if ( !strcmp (name [i], "bdata") )
mir->bdata = g_strdup(cols [i]);
+ /* Evolution itself doesn't yet use this, ignoring
+ else if ( !strcmp (name [i], "bodystructure") )
+ mir->bodystructure = g_strdup(cols [i]); */
}
}
@@ -2063,7 +2066,7 @@ camel_folder_summary_add_from_parser(CamelFolderSummary *s, CamelMimeParser *mp)
CamelMessageInfo *
camel_folder_summary_add_from_message (CamelFolderSummary *s, CamelMimeMessage *msg)
{
- CamelMessageInfo *info = camel_folder_summary_info_new_from_message(s, msg);
+ CamelMessageInfo *info = camel_folder_summary_info_new_from_message(s, msg, NULL);
camel_folder_summary_add (s, info);
update_summary (s, (CamelMessageInfoBase *) info);
@@ -2160,6 +2163,7 @@ camel_folder_summary_info_new_from_parser(CamelFolderSummary *s, CamelMimeParser
* camel_folder_summary_info_new_from_message:
* @summary: a #CamelFodlerSummary object
* @message: a #CamelMimeMessage object
+ * @boydstructure: a bodystructure or NULL
*
* Create a summary item from a message.
*
@@ -2167,13 +2171,13 @@ camel_folder_summary_info_new_from_parser(CamelFolderSummary *s, CamelMimeParser
* #camel_message_info_free
**/
CamelMessageInfo *
-camel_folder_summary_info_new_from_message(CamelFolderSummary *s, CamelMimeMessage *msg)
+camel_folder_summary_info_new_from_message(CamelFolderSummary *s, CamelMimeMessage *msg, const gchar *bodystructure)
{
CamelMessageInfo *info;
struct _CamelFolderSummaryPrivate *p = _PRIVATE(s);
CamelIndexName *name = NULL;
- info = ((CamelFolderSummaryClass *)(CAMEL_OBJECT_GET_CLASS(s)))->message_info_new_from_message(s, msg);
+ info = ((CamelFolderSummaryClass *)(CAMEL_OBJECT_GET_CLASS(s)))->message_info_new_from_message(s, msg, bodystructure);
/* assign a unique uid, this is slightly 'wrong' as we do not really
* know if we are going to store this in the summary, but we need it set for indexing */
@@ -2975,11 +2979,12 @@ content_info_new_from_parser(CamelFolderSummary *s, CamelMimeParser *mp)
}
static CamelMessageInfo *
-message_info_new_from_message(CamelFolderSummary *s, CamelMimeMessage *msg)
+message_info_new_from_message(CamelFolderSummary *s, CamelMimeMessage *msg, const gchar *bodystructure)
{
CamelMessageInfo *mi;
mi = ((CamelFolderSummaryClass *)(CAMEL_OBJECT_GET_CLASS(s)))->message_info_new_from_header(s, ((CamelMimePart *)msg)->headers);
+ ((CamelMessageInfoBase *)mi)->bodystructure = g_strdup (bodystructure);
return mi;
}
@@ -3201,6 +3206,9 @@ message_info_from_db (CamelFolderSummary *s, CamelMIRecord *record)
mi->cc = (gchar *) camel_pstring_add (record->cc, FALSE);
mi->mlist = (gchar *) camel_pstring_add (record->mlist, FALSE);
+ /* Evolution itself doesn't yet use this, so we ignore it (saving some memory) */
+ mi->bodystructure = NULL;
+
/* Extract Message id & References */
mi->content = NULL;
part = record->part;
@@ -3387,6 +3395,8 @@ message_info_to_db (CamelFolderSummary *s, CamelMessageInfo *info)
record->followup_completed_on = (gchar *) camel_pstring_strdup(camel_message_info_user_tag(info, "completed-on"));
record->followup_due_by = (gchar *) camel_pstring_strdup(camel_message_info_user_tag(info, "due-by"));
+ record->bodystructure = mi->bodystructure ? g_strdup (mi->bodystructure) : NULL;
+
tmp = g_string_new (NULL);
if (mi->references) {
g_string_append_printf (tmp, "%lu %lu %lu", (gulong)mi->message_id.id.part.hi, (gulong)mi->message_id.id.part.lo, (gulong)mi->references->size);
diff --git a/camel/camel-folder-summary.h b/camel/camel-folder-summary.h
index 6281d18..560a882 100644
--- a/camel/camel-folder-summary.h
+++ b/camel/camel-folder-summary.h
@@ -188,7 +188,7 @@ struct _CamelMessageInfoBase {
CamelMessageContentInfo *content;
struct _camel_header_param *headers;
gchar *preview;
-
+ gchar *bodystructure;
};
/* probably do this as well, removing CamelFolderChangeInfo and interfaces
@@ -275,7 +275,7 @@ struct _CamelFolderSummaryClass {
/* create/save/load an individual message info */
CamelMessageInfo * (*message_info_new_from_header)(CamelFolderSummary *, struct _camel_header_raw *);
CamelMessageInfo * (*message_info_new_from_parser)(CamelFolderSummary *, CamelMimeParser *);
- CamelMessageInfo * (*message_info_new_from_message)(CamelFolderSummary *, CamelMimeMessage *);
+ CamelMessageInfo * (*message_info_new_from_message)(CamelFolderSummary *, CamelMimeMessage *, const gchar *);
CamelMessageInfo * (*message_info_load)(CamelFolderSummary *, FILE *);
gint (*message_info_save)(CamelFolderSummary *, FILE *, CamelMessageInfo *);
gint (*meta_message_info_save)(CamelFolderSummary *, FILE *, FILE *, CamelMessageInfo *);
@@ -380,7 +380,7 @@ CamelMessageInfo *camel_folder_summary_add_from_message(CamelFolderSummary *summ
/* Just build raw summary items */
CamelMessageInfo *camel_folder_summary_info_new_from_header(CamelFolderSummary *summary, struct _camel_header_raw *headers);
CamelMessageInfo *camel_folder_summary_info_new_from_parser(CamelFolderSummary *summary, CamelMimeParser *parser);
-CamelMessageInfo *camel_folder_summary_info_new_from_message(CamelFolderSummary *summary, CamelMimeMessage *message);
+CamelMessageInfo *camel_folder_summary_info_new_from_message(CamelFolderSummary *summary, CamelMimeMessage *message, const gchar *bodystructure);
CamelMessageContentInfo *camel_folder_summary_content_info_new(CamelFolderSummary *summary);
void camel_folder_summary_content_info_free(CamelFolderSummary *summary, CamelMessageContentInfo *ci);
diff --git a/camel/providers/groupwise/camel-groupwise-journal.c b/camel/providers/groupwise/camel-groupwise-journal.c
index 6e34e36..1068b34 100644
--- a/camel/providers/groupwise/camel-groupwise-journal.c
+++ b/camel/providers/groupwise/camel-groupwise-journal.c
@@ -367,7 +367,7 @@ update_cache (CamelGroupwiseJournal *groupwise_journal, CamelMimeMessage *messag
camel_object_unref (cache);
- info = camel_folder_summary_info_new_from_message (folder->summary, message);
+ info = camel_folder_summary_info_new_from_message (folder->summary, message, NULL);
camel_pstring_free(info->uid);
info->uid = camel_pstring_strdup (uid);
diff --git a/camel/providers/groupwise/camel-groupwise-summary.c b/camel/providers/groupwise/camel-groupwise-summary.c
index fe1273e..b765645 100644
--- a/camel/providers/groupwise/camel-groupwise-summary.c
+++ b/camel/providers/groupwise/camel-groupwise-summary.c
@@ -427,7 +427,7 @@ camel_gw_summary_add_offline (CamelFolderSummary *summary, const gchar *uid, Cam
const CamelTag *tag;
/* Create summary entry */
- mi = (CamelGroupwiseMessageInfo *)camel_folder_summary_info_new_from_message (summary, message);
+ mi = (CamelGroupwiseMessageInfo *)camel_folder_summary_info_new_from_message (summary, message, NULL);
/* Copy flags 'n' tags */
mi->info.flags = camel_message_info_flags(info);
diff --git a/camel/providers/imap/camel-imap-folder.c b/camel/providers/imap/camel-imap-folder.c
index 6cc8cca..18fcdd9 100644
--- a/camel/providers/imap/camel-imap-folder.c
+++ b/camel/providers/imap/camel-imap-folder.c
@@ -3239,6 +3239,7 @@ add_message_from_data (CamelFolder *folder, GPtrArray *messages,
CamelStream *stream;
CamelImapMessageInfo *mi;
const gchar *idate;
+ const gchar *bodystructure;
gint seq;
seq = GPOINTER_TO_INT (g_datalist_get_data (&data, "SEQUENCE"));
@@ -3257,7 +3258,9 @@ add_message_from_data (CamelFolder *folder, GPtrArray *messages,
return;
}
- mi = (CamelImapMessageInfo *)camel_folder_summary_info_new_from_message (folder->summary, msg);
+ bodystructure = g_datalist_get_data (&data, "BODY");
+
+ mi = (CamelImapMessageInfo *)camel_folder_summary_info_new_from_message (folder->summary, msg, bodystructure);
camel_object_unref (CAMEL_OBJECT (msg));
if ((idate = g_datalist_get_data (&data, "INTERNALDATE")))
@@ -3413,7 +3416,7 @@ imap_update_summary (CamelFolder *folder, gint exists,
size = (exists - seq) * (IMAP_PRETEND_SIZEOF_FLAGS + IMAP_PRETEND_SIZEOF_SIZE + IMAP_PRETEND_SIZEOF_HEADERS);
got = 0;
if (!camel_imap_command_start (store, folder, ex,
- "UID FETCH %d:* (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[%s])",
+ "UID FETCH %d:* (FLAGS RFC822.SIZE INTERNALDATE BODYSTRUCTURE BODY.PEEK[%s])",
uidval + 1, header_spec->str)) {
g_string_free (header_spec, TRUE);
return;
@@ -3499,7 +3502,7 @@ imap_update_summary (CamelFolder *folder, gint exists,
while (uid < needheaders->len && !camel_application_is_exiting) {
uidset = imap_uid_array_to_set (folder->summary, needheaders, uid, UID_SET_LIMIT, &uid);
if (!camel_imap_command_start (store, folder, ex,
- "UID FETCH %s BODY.PEEK[%s]",
+ "UID FETCH %s BODYSTRUCTURE BODY.PEEK[%s]",
uidset, header_spec->str)) {
g_ptr_array_free (needheaders, TRUE);
camel_operation_end (NULL);
diff --git a/camel/providers/imap/camel-imap-summary.c b/camel/providers/imap/camel-imap-summary.c
index fc100b0..1492296 100644
--- a/camel/providers/imap/camel-imap-summary.c
+++ b/camel/providers/imap/camel-imap-summary.c
@@ -454,7 +454,7 @@ camel_imap_summary_add_offline (CamelFolderSummary *summary, const gchar *uid,
const CamelTag *tag;
/* Create summary entry */
- mi = (CamelImapMessageInfo *)camel_folder_summary_info_new_from_message (summary, message);
+ mi = (CamelImapMessageInfo *)camel_folder_summary_info_new_from_message (summary, message, NULL);
/* Copy flags 'n' tags */
mi->info.flags = camel_message_info_flags(info);
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]