EggDBus

[David Zeuthen <david fubar dk>]

Hey,

For the past 5 weeks or so, I've been working on a new (as compared to
dbus-glib) D-Bus binding for GObject. The work on this has finally
reached a stage where the code sufficiently complete and documented so I
thought I'd send some mail describing it. The code is here

 http://cgit.freedesktop.org/~david/eggdbus

First, let me briefly describe the motivation behind doing this work.
Well, there's a couple of reasons:

 o  The venerable dbus-glib bindings doesn't really make it easy (nor
    enjoyable) to use D-Bus from GObject (especially not C). At the same
    time other runtimes, such as Qt, have much more complete D-Bus
    bindings.

 o  I was at a point where I needed to either implement or use (for
    most of them, both) a rather large set of complex D-Bus services:
    PolicyKit rewrite, ConsoleKit, DeviceKit, DeviceKit-disks,
    DeviceKit-power.

    The time it would take to make proper glue code (e.g. GObject
    properties, signals, encapsulations, async+sync methods etc.) for
    GObject based programs would be on the order of several months.
    That is, if I hadn't gone postal while doing it. Thus: EggDBus.

First, I defined the goals

 1. The IPC aspect of D-Bus should be as transparent as possible; e.g.
    where possible D-Bus concepts should be mapped to something the
    programmer is familiar with. For example, passing an array of
    structures shouldn't mean messing around with GValues (I regard
    GValue as an implementation detail of GObject; YMMV).

 2. The life cycle aspect of D-Bus (which is often an ignored feature
    of D-Bus: see http://cgwalters.livejournal.com/16885.html ) should
    be very easy to use as well. In other words, it should be very easy
    to determine if someone already owns a Name on the bus; get notified
    when the name changes and so on.

    (Incidentally, a large number of people gets it wrong (I've gotten
    it wrong many times myself; not pointing fingers here) and their
    code ends getting woken up whenever a new name appears or disappears
    on the bus. So it would seem prudent to have good library support
    for this kind of thing.)

 3. Easy of use, especially from C, is important. E.g. it should be
    possible to define an interface in an IDL like language (more on
    that below), run a code generator, and then, presto, easily consume
    or provide a D-Bus service. This includes having things like easily
    accessible docs for the generated glue code.

    (The desired work-flow for dynamic languages is a bit different; I'm
     focusing on static languages for now)

Then I took a good look at what's out there. I looked a lot at Telepathy
which has a really neat partial solution for goal 1: GInterfaces.
However, it's still (at least I believe) a shim on top of dbus-glib's
GValue handling. And AFAICT D-Bus properties aren't mapped to GObject
properties. Also, the library isn't yet separated out; there's a bunch
of Telepathy specific bits in telepathy-glib.

On to how EggDBus works.

First, you define the D-Bus interfaces in an "IDL language". Here's an
example from the EggDBus test suite

 (Note: the "IDL language" is presently annotated D-Bus introspection
  XML. Yes, I'd be the first to say this is not ideal. In fact, it's
  fucking ugly. FWIW, I have plans to define a real IDL language in
  cooperation with other bindings authors in the D-Bus community.)

http://cgit.freedesktop.org/~david/eggdbus/tree/src/tests/com.example.Frob.xml?id=0.1
http://cgit.freedesktop.org/~david/eggdbus/tree/src/tests/com.example.Tweak.xml?id=0.1
http://cgit.freedesktop.org/~david/eggdbus/tree/src/tests/com.example.Twiddle.xml?id=0.1

In the EggDBus Introspection XML IDL, you define

 o interfaces
   - methods
   - properties
   - signals
 o error domains
 o flag enumerations
 o enumerations
 o documentation
 o structures

 (The latter five items are not defined by the D-Bus specification but
  it's implicitly used anyway and it's hard not to argue that these
  items are not part of an interface. So it should be defined in the
  same place. That way you won't have to change tons of code whenever
  a new error or flag is introduced.)

>From the IDL we now generate C code. This code has gtk-doc comments
(generated from the IDL) so instead of showing you the code, it's easier
just to point to the generated gtk-doc HTML (the section "Example of
Generated Code"):

http://people.freedesktop.org/~david/eggdbus-0.1-docs/

Here are the highlights

 o Each D-Bus interface maps to a GInterface
   - each D-Bus method maps to
     - three client side functions, e.g.
       - test_frob_hello_world_sync()
       - test_frob_hello_world()
       - test_frob_hello_world_finish()
     - one server side VTable entry, e.g.
       (*handle_hello_world)()
     - one server side function
       test_frob_handle_hello_world_finish()

   - each D-Bus signal maps to
     - A GObject signal, e.g.
       - com.example.Twiddle.NewzNotifz -> TestTwiddle:newz-notifz
     - A type-safe convenience function for emitting the signal
       - test_twiddle_emit_signal_newz_notifz()

   - each D-Bus property maps to
     - A GObject property, e.g.
       - com.example.Tweak.SomeReadWritePropety -> 
         TestTweak:some-read-write-property
       - C getters and setters (depending on the property access flags)

   - examples:
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/tests-testfrob.html
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/tests-testtweak.html
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/tests-testtwiddle.html

 o Each D-Bus structure maps to a EggDBusStructure derived class
   - with getters/setters for each element
   - examples:
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/TestPoint.html
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/TestStructWithVariant.html

 o Each error domain maps to a generated GError error domain
   - examples:
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/tests-testerror.html
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/tests-testdetailederror.html

 o Flags, enumerations maps to generated subtypes of GEnum and GFlags
   - examples:
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/tests-testcreateflags.html
     http://people.freedesktop.org/~david/eggdbus-0.1-docs/tests-testvehicle.html

Note that the standard D-Bus interfaces provided in EggDBus are also
generated from annotated IDL, e.g.

http://people.freedesktop.org/~david/eggdbus-0.1-docs/eggdbus-eggdbuspeer.html

is generated from

http://cgit.freedesktop.org/~david/eggdbus/tree/src/eggdbus/org.freedesktop.DBus.Peer.xml?id=0.1

Now, onto how this is used. It's very simple. On the client side you use
the generated functions (such as test_frob_hello_world()) while on the
server side you implement an object that implements the GInterface in
question. I'm not going to paste code here, look at the test suite
instead:

http://cgit.freedesktop.org/~david/eggdbus/tree/src/tests/testclient.c?id=0.1
http://cgit.freedesktop.org/~david/eggdbus/tree/src/tests/testserver.c?id=0.1
http://cgit.freedesktop.org/~david/eggdbus/tree/src/tests/testfrobimpl.c?id=0.1
http://cgit.freedesktop.org/~david/eggdbus/tree/src/tests/testtweakimpl.c?id=0.1
http://cgit.freedesktop.org/~david/eggdbus/tree/src/tests/testtwiddleimpl.c?id=0.1

Note that all server methods are async by default. You could even do
things like run each method invocation in a separate thread or whatever.

For name tracking, there's an extremely simple (and I think, useful) way
to track names

http://people.freedesktop.org/~david/eggdbus-0.1-docs/EggDBusObjectProxy.html#EggDBusObjectProxy--name-owner

Now, in order to handle complex data types I really needed something
like libgee since I need to know the type of the data in order to shove
it onto the wire (via libdbus). 

Initially I just did things like using my own boxed GHashTable type that
stored the D-Bus signature / GType in a lookaside data structure. It was
still very painful to use from C.

So, for some historical context, there's already a push to get interface
collections into the GLib stack

http://bugzilla.gnome.org/show_bug.cgi?id=560061

AFAICT, the main sticking point here is C convenience (especially
wanting allocating iterators on the stack) and after having experimented
with libgee I tend to concur.

So I ended up writing my own EggDBusArraySeq and EggBusHashMap classes
to make it more friendly to use from C (they actually make it easy to
use fixed-size types, something that the GLib data types don't). 

While these classes of mine don't yet implement any of the proposed
collection interfaces (such as GSeq or GeeList) they are very much
designed to be able to do so. I will follow up on that bug with more
detailed thoughts.

For mapping D-Bus structures to the GObject, I ended up doing something
that is possibly controversial. The story is here

http://people.freedesktop.org/~david/eggdbus-0.1-docs/EggDBusStructure.html#EggDBusStructure.description

Depending on who you ask this could either be clever use of the GObject
type system or Something That Shouldn't Be Done(tm). Either way, I don't
see any good alternative if you want a very simple and user-friendly
API.

Some other notes about EggDBus

 - It's going to be used in a number of projects of mine; it's not
   yet production ready but I expect that to change over the next
   few months...

 - Is using libdbus-1 as an _implementation_ detail. E.g. if we decided
   to change to a more light-weight D-Bus implementation (say, with
   a GLib-ish abort-on-OOM handling) this can be done behind the scenes.

 - Is designed specifically to integrate with the GObject type system.

 - Is designed to be able to handle both arbitrary complex data types
   - if you follow the dbus list you will note that every few months
     there's some poor person who run into dbus-glib limitations when
     trying to pass complex data types around.

 - Correctness is a concern so there's a very comprehensive test suite
   already.

 - Easy of use is a major concern so there's already good doc coverage
   and a tutorial planned.

 - Like D-Bus, EggDBus is not a panacea. It is designed specifically
   to make it comfortable to work with very large and potentially
   complex D-Bus services like e.g. DeviceKit-disks:
 
   http://hal.freedesktop.org/docs/DeviceKit-disks/Device.html
   (we expect to add a lot more D-Bus interfaces to support things
    like LVM, btrfs etc. in the near future. So the interfaces are
    going to grow).

   from C and GObject. Specifically, EggDBus is not designed for
   raw throughput; if you need that you probably shouldn't be using
   D-Bus in the first place.

   Anyway, without having measured anything there's going to be *some*
   overhead in using EggDBus just compared to libdbus-1. But this is
   *fine* for most services; current D-Bus services we have don't need
   throughput.

   (case in point: I think I agree with Ryan that things like GSettings
    shouldn't be using D-Bus for reading properties since you're going
    to have 20+ apps all doing that on startup.)

 - Might look over-engineered. It might. But keep in mind it's designed
   to a) be easy to use; and b) scale to very large and complex D-Bus
   services. 

 - Performance is a actually a concern especially when it comes to
   avoiding wakeups. And of course, avoiding burning CPU cycles
   is of course a concern (e.g. use O(1) algorithms where possible).

   Wrt. performance, some people may be concerned about the extensive
   use of classes and GObject features.  Either way, I think that
   alexl's work on GIO (EggDBus is comparable to GIO in complexity and
   size; perhaps a bit simpler / smaller) have shown that just because
   you're using GObject doesn't mean you have to be slow.

 - Is designed for C/C++ but written in a way such that the core bits
   should be useful in language bindings (for example, there's an
   introspection parser in EggDBus).

 - Is still a work in progress. See

   http://cgit.freedesktop.org/~david/eggdbus/tree/docs/TODO?id=0.1

   There's also a bunch of TODO items in the code. Nothing major (mostly
   optimizations that can be done), just saying. Nothing is set in stone
   yet either.

This mail is already too long but there's one more thing.

The reason I chose to call this library EggDBus is because I think it
would make sense to merge something like this into GLib; here's why

 - The use of D-Bus in the desktop space is already huge and
   it's growing. We really need something that is easier to use
   than dbus-glib. Right now there's no good answer in GNOME for
   this if you're writing C code.

 - It would be nice to be able to D-Bus from GTK+. Things like
   GUniqueApplication comes to mind.

If there's interest in this I'd be happy with doing the work necessary
to make it happen (including ensuring that D-Bus on Win32 works etc.).
If not, I'll just rename EggDBus to something else...

Finally, sorry for writing such a long mail. FWIW, I realize that many
people on this list are not necessarily familiar with (or interested in)
all the D-Bus concepts and specifics covered in this mail. Nonetheless,
I hope it's clear that it would be a good thing if we had a good answer
for D-Bus in the core GNOME library stack.

(And if you got this far, I'll buy you a beer at GUADEC ;-)

Thanks,
David