Online Desktop, Tomboy, and user storage (was Re: Dogfood servers now up)

["Sanford Armstrong" <sanfordarmstrong gmail com>]

On 8/1/07, Havoc Pennington <hp redhat com> wrote:
> Hi,
>
> Sanford Armstrong wrote:
> > How does this fit into the grand scheme of the online desktop?  Tomboy
> > can't be the only app that wants to store real data (not just
> > configuration data) out there.  I know Mugshot is meant mostly to bind
> > existing services together, but how is a small free software project
> > supposed to find the resources to offer a stable and performant web
> > service to a large number of users?  The answer for the Online Desktop
> > can't always be "store your data in an existing cool (proprietary)
> > service".  GNOME is going to have to become a service provider.
>
> I agree, the only way to make service-provider-dependent features usable
> is to have at least a default provider available, though people can also
> host their own.
>
> What is your order of magnitude on the size of someone's Tomboy notes?
> If they aren't very big then let's just try to get it going now. I can
> help on the server side if you can point me to some sense of what the
> data looks like, relevant source code, etc.

 * * * Warning, boring details follow: * * *

My test instance of Tomboy is 96 notes; the average Tomboy user
probably has less than 500 notes.  On the server where I am
synchronizing those notes, I am using 432 KB.  So I think 10 MB would
be sufficient for most Tomboy users.

Each note is stored in a separate .note file, which is just XML
(typically 0.6-4.0 KB).

For our file system sync backend, typical synchronization server
layout looks like this (semi-inspired by svn repository layout, though
don't be fooled...we are only storing the most recent revision for
each note, there is no saved history):

server-root/
  manifest.xml - (maps notes to revision numbers)
  0/ - (contains directories for revisions 0-99)
    0/ - (contains notes from revision 0)
      x.note
      y.note
    1/ - (contains notes from revision 1)
      z.note
    ...
  1/ - (contains directories for revisions 100-199)
    0/ - (contains notes from revision 100)
  ...

A user can point to any path on their file system to use as a "server"
(an nfs or smb mount, for example), or for WebDAV and SSH we automate
the process using FUSE file systems wdfs and sshfs.  The architecture
for building sync service backends is pretty flexible, and third
parties can develop their own using our addin architecture.

> Maybe we should also consider a web page to access your notes, which
> would let you get to them from a random internet kiosk and the like.

This would be awesome and I have some sketches of what this might look
like (plus we already have xsl transforms for exporting notes to
HTML).  But I don't think we have the resources to work on this before
the end of the current cycle.

> For a small quota we can support quite a few users without too much
> trouble. Think a reasonable hard drive (200G) divided by a 10M quota.

As mentioned above, that should be quite reasonable.

> Something to consider, on the server we can store files or we can store
> database tables. They both have some advantages, with the database
> tables we can do fine-grained change notification over XMPP and it's
> easier to support a web view and web editing. With files it's easier to
> deal with "large" (exact definition fuzzy) data.

I need to read more about the data model design in Mugshot.  But I can
say that with regard to this cycle, the Tomboy team can't add support
for any new sync backends that don't have a corresponding FUSE file
system.  If online.gnome.org can provide WebDAV access, we can very
easily add support.  Otherwise, we'll have to wait until September to
start work on it.  Of course, patches are always welcome.  ;-)


So to sum up, if Mugshot can provide free WebDAV shares for each user,
then the basic needs are set for Tomboy synchronization.  Awesome
ideas for the future:

* Web app for viewing notes, making notes public, viewing friends'
notes, printing notes, and downloading arbitrary notes for inclusion
on your local Tomboy instance
* More generic API for note data transfer, so server can store data
however it likes and user doesn't have to worry about installing wdfs,
configuring FUSE, etc.

These are all things I'm psyched to work on; just let me know what I
can do to help!

Sandy