Re: API proposal for charset code conversion at I/O

[Owen Taylor <otaylor redhat com>]

Sven Neumann <sven gimp org> writes:

> Hi,
> 
> Owen Taylor <otaylor redhat com> writes:
> 
> > This is a forward of some notes from a discussion with HideToshi
> > Tajima from Sun about what would be necessary to add a useful
> > streaming charset conversion API to GIOChannel.
> 
> > Now, long-term I'd like to see GIOChannel converted into something
> > much nice more full-featured. Along the lines of the 
> > Java character/byte streams, or Qt's QTextStream. 
> 
> Shouldn't that be a layer build on top of GIOChannels? IMHO GIOChannels
> provide a useful abstraction from UNIX sockets and pipes. Adding buffered
> IO adds an unnecessary level of complexity. 

My attempt was to make the minimal API changes necessary for charset
conversion, not create a lot of new API elaboration, which, I think
might be best done by simply replacing GIOChannel with something
better. Looking back on my design for GIOChannel a few years later,
there are a lot of things I think I did wrong:

 - It's API is too tied to that of Unix read/write
 - The error reporting is primitive
 - It doesn't retry reads/writes
 - It isn't actually convenient for much of anything


If you look at my proposed API revision, you'll see that the buffering
is all done in the generic code; specific subclasses of GIOChannel
dont' have to worry about it, and there would still be the possiblity
of doing raw reads/writes the old way.


The API's I'm aware of in this area include, among others:

 QTextStream

  http://doc.trolltech.com/qtextstream.html

 .NET System.IO

  http://msdn.microsoft.com/library/dotnet/cpref/frlrfSystemIO.htm

 Java character streams

  http://java.sun.com/j2se/1.3/docs/guide/io/io.html

(Another thing to look at would be the APR interface in this area,
though I couldnt' quickly find docs for that; Also it might
be worth looking at the streaming interfaces in Bonobo/gnome-vfs.)

All three of these do make some distinction between a low level
layer and a higher level layer, with character set conversion
living in the higher layer. 

But both QIODevice and System.IO.Stream (the low level layers)
actually have buffering in the low level layer.

It's quite possible we should have some such low-level / high-level
distinction eventually.


But I don't think we have time to figure out the best possible API for
GLib-2.0. So if we want charset conversion, we need to think about
the the minimal changes that would be necessary to do that.

I think a introducing a double layered scheme without fully
exploring the exact details necessary would give us a lot
more legacy baggage to deal with later than simply extending
GIOChannel a bit.

 
> > GIOChannel needs to have buffering, and a representation
> > of blocking/non-blocking IO.
> 
> We (convergence integrated media GmbH) have an IO library build on 
> top of GIOChannels that provides buffering and asynchronous operation 
> in the GMainLoop. If you are interested, contact me and I will make 
> that code available to you. It's scheduled for release under the LGPL 
> anyway.

Well, I think people would be interested in seeing the API for that
at least, so if you could post it here, that would be interesting.

Regards,
                                        Owen