Re: GVariant for prez!

[Simon McVittie <simon mcvittie collabora co uk>]

On Sat, 02 May 2009 at 13:02:39 -0400, David Zeuthen wrote:
>    My view is that it's not really realistic to do this, I think it's
>    a convenient thing that I can get/pass well-known data-types like
>    GHashTable or GPtrArray from/to remote objects. This allows adding
>    D-Bus support to existing programs without fundamentally changing
>    how their data is represented and not having to deal with a vararg
>    based API should you choose to generate code.

I don't think producing a high-quality D-Bus API from an existing C API is
as realistic or useful as you think. A D-Bus API is an external interface,
just like a C API, a file format or a network protocol, and as such it needs
to be designed (and as in file formats and network protocols, there are
conventions you can use, like DBus.Properties, but you still have to know how
those conventions are applied to this particular use).

For an extreme case of "my D-Bus API is my C API" see Pidgin's auto-generated
D-Bus API, which IIRC can be segfaulted remotely by passing an inappropriate
handle-representing-a-pointer to a method!

Most of the Telepathy stack consists of code that was explicitly written for
dbus-glib use, and even there, there are fewer situations than you might think
where we can push an internal data structure straight into dbus-glib. In
practice there's a lot of "take this list of GObjects, call this method on
each of them, put the results in a GPtrArray and hand it to dbus-glib". It'd
be no harder for us to put the results in a GVariant and hand *that* to a
D-Bus binding.

If a binding for a particular protocol wants to map away from the data types
of the wire protocol into types that make more sense for the situation, that's
fine, but I don't think it's useful to do this in a generic way without
knowledge of the particular API.

If you're generating code, then presumably the code generator is verifiably
correct, so it doesn't matter how error-prone the API it calls into is - in
telepathy-glib, a significant amount of our code generation exists to call
into APIs that are only available as varargs, either adding type-safety in a
wrapper function or doing marshalling/unmarshalling to work around dbus-glib.
As long as the code generator itself is sufficiently well-tested (by which I
mean regression tests with a "one of each type" approach, to check that the
author of the code generator didn't get it wrong), it won't make mistakes.

>    It also seems particularly awkward to not map D-Bus APIs to native
>    APIs (in GLib, GptrArray, GArray and GHashTable are considered native
>    APIs for collections) - by the same token, would you expect Python
>    users to use GVariant? No, they'd want to use the native types so
>    they have things like hash table literals etc.

*dbus-python hat on*

Actually, if I had time to work on dbus-python, the API would become more like
GVariant, and the internal representation would certainly be GVariant-style
serialization-in-advance (I've thought about this a lot, independently of
Ryan's GVariant work). There'd be some sort of Message, Struct, Array, Dict
types, whose API resembled Python lists/dicts/etc. but which were *not*
subclasses of list/dict/etc. like the current API; instead of storing their
data according to the superclass' data model, they'd serialize it into
fragments of D-Bus message straight away.

The current dbus-python API jumps through hoops to try to be "just like normal
Python", with the result that trivial uses of dbus-python are trivial, and
non-trivial uses of dbus-python are error-prone and can't be done correctly
without quite a lot of knowledge about D-Bus, Python, and dbus-python
implementation details.

Optimizing for trivial uses was a mistake - you have to know at least the
basics of how D-Bus types work if you're going to have any chance at all of
interoperating with a different language, binding or maintainer, and if that
knowledge can be assumed, then we can use it to make a much less error-prone
and more consistent API.

Using serialization-in-advance like GVariant would mean that if you are able
to construct (say) a Struct without it raising an exception, then sending that
Struct over D-Bus cannot fail (in the current dbus-python, programmer error
only becomes visible at the time the message is to be sent, which is too late),
and indexing into that Struct returns exactly what the recipient would see (as
opposed to what you put in, which might have been type-transformed - e.g. if
you put a subtype of str into a current dbus-python Array, then index into it,
you get your custom subtype, but that's not what the recipient will get).

Of course, there'd be an easy constructor for Dict that took a Python dict
as argument, maybe Dict('sv', {'badger': 'mushroom'}) or something, and the
normal Python syntax like my_dbus_dict['badger'] would work on it, but Dict
would not be a subtype of the built-in dict.

Similarly, Array would be a sequence (thing that supports a[0] syntax)
and an iterable (thing that supports "for ... in ..." loops), and would have a
constructor that took a D-Bus type and an iterable, but it would not be a
subtype of list like it is now.

The fact that GVariant instances are always immutable is something that hadn't
occurred to me, but now that I think about it, I believe it to be correct.

>    Specifically, there is no equivalent of g_hash_table_lookup() etc.
>    right now on GVariant but I suppose you can add that.

I suggested a fairly efficient implementation on the bug (O(n) setup the first
time it's used, and the same complexity as GHashTable thereafter).

>    (Btw, if GVariant was based on GObject and we added collection
>     interfaces to the GLib stack (like some people want, cf. bug
>     560061), then you'd just make GVariant implement the these
>     interfaces and that would solve the "there is no
>     g_hash_table_lookup() equivalent" complaint.)

Right, this is how I'd like dbus-python to work - Array shouldn't be a subtype
of list, but thanks to duck typing, Array *would* implement the ad-hoc
"sequence interface".

>    I realize that some people are not opposed to the GVariant type
>    of programming. E.g. build your data up basically using an iterator,
>    much like the way you build a DBusMessage. Obviously, I can't speak
>    for others, but I personally don't like that kind of API unless I'm
>    tooling around with simple D-Bus services. YMMV.

Building messages in GVariant doesn't look much like DBusMessage to me:
see http://git.gnome.org/cgit/glib/commit/?h=gvariant&id=00ddd213b817 for
instance. What is it that you object to?

>  o I'm worried that GVariant supports a superset of the D-Bus
>    protocol.

Right, ideally GVariant as merged in GLib should support exactly the D-Bus
protocol (either by code deletion, #if 0, or getting more things into the D-Bus
protocol).

>  o I don't see any tests cases at all (or am I looking in the wrong
>    place) - this worries me.

See link above.

>  o I don't really like the name, I don't think it adequately
>    describes what GVariant is: a serialization format with an API
>    to access and build the value. So, I'd call it GDBusValue instead
>    since that really is what it is.

I'd be inclined to give it a less D-Bus name, since its other use is as
(a piece of data in) an mmap'able binary equivalent of GKeyFile - Havoc
suggested some good possibilities although they're all rather long and
unwieldy. It's a pity GData already exists, since that'd be a good name for
this concept.

GStored? GStorage? GRepresentation? (OK, that one's too long.) GBlob?

Would the 2-dimensional-database connotations make GRecord too misleading?
Unfortunately, I think they would - people would get confused by a record
containing more records.

Alternatively, thinking of a GVariant as a piece of a larger collection of data
suggests words like snippet, chunk, morsel.

(Every programmer needs a thesaurus :-)

>  o The C API for invoking methods / emitting signals would also accept
>    G_TYPE_DBUS_VALUE. This, basically, allows you to feed raw D-Bus
>    messages.

If messages are represented internally as GVariant anyway, why wouldn't the
lower-level C API be in those terms? Generated or wrapper code could still do
whatever type-mapping it wanted to.

> Of course all this is up to the GLib maintainers (maybe they'll merge
> GVariant, maybe they'll merge GDBus, maybe they'll merge nothing!) but I
> don't think landing both GDBus and GVariant without some kind of
> integration work (perhaps like what I proposed) is ever going to work.
> It would be truly bizarre.

Right, having a D-Bus message passing API and an API for the D-Bus data model,
but not having the former use the latter, would seem perverse.

    Simon