Re: State of gdk-pixbuf

[Bastien Nocera <hadess hadess net>]

On Fri, 2014-10-24 at 13:30 +0000, Benjamin Otte wrote:
> Bastien Nocera <hadess <at> hadess.net> writes:
>
> > I'm particularly interested to know what cairo, pixman and other image
> > manipulation libraries can do for us. Benjamin surely has comments[2] :)
> I think gdk-pixbuf should just die already, but it won't. And now you're
> resurrecting it, dammit!

While there's nothing to replace it, probably not. And so far, I'm
mainly wrangling patches and fixing small bugs.

> The reason is that I think that gdk-pixbuf is both a rather old library so
> it has very sucky conventions in lots of places (like exposing way too many
> externals) and that it isn't a good fit for the use cases we have.
> But it's ubiquitous and easy to use when it does work, so it's always the
> place people go to.

Exposing too many internals? Or do you mean its API is too wide?

> I'll try to elaborate on the use cases we have and why it's not been
> replaced for each of those
>
> 1. image loading
> gdk-pixbuf provides a generic way to hand it a file or stream and get pixels
> out the other end. There is no other way to do this, unless you are a
> masochist and want to use the image loading libs directly. The loaders in
> gdk-pixbuf have also been sanitized well enough to be safe (enough) against
> random data from the Internet.
> The reason gdk-pixbuf is used here is because there's no replacement that is
> as easy to use and integrates as well into the Gnome stack. Webkit has its
> own loaders, but they are internal to Webkit (I'd expect them to be even
> safer). Gimp/Gegl has its own loaders, but they pull in libgimp (or you need
> to code your own widgets to display them).

OK, so we agree that we should be keeping that part at least. If I
remember correctly, fixing potential security problems in the loaders is
how Matthias got started ;)

We could probably do with a fuzzer and throwing images at it, or with a
couple of coverity runs.

> 2. holding pixel data
> A GdkPixbuf is a kind of agnostic way to hold a (width, height, contents)
> object. It's rather ad at this because it uses the worst format possible
> (unpremultiplied RGBA in the wrong order). 
> GTK is slowly replacing its internal APIs with cairo_surface_t here. And I
> expect this to potentially change again to some sort of texture thing once
> we move towards GL.

I've been looking at a way to make this more efficient, and babl might
be a good way to store decoded data in a format that's closer to the
format used internally. Eg. a grayscale picture shouldn't take as much
room in memory as a same-dimension colour one.

Once you actually want to display, you'd need to convert it to a format
cairo understands. We have some pixbuf/cairo APIs in GDK, should those
be in gdk-pixbuf instead?

> 3. an area of custom drawing
> GTK kindly kindly provides lots of API to use GdkPixbuf everywhere - from
> gdk_cairo_set_source_pixbuf() to lots of widgets that can display it
> (GtkImage, GtkEntry, GtkClipboard, GdkCellRendererPixbuf...). So people use
> pixbufs whenever they have some custom drawing that they want displayed
> (like icons, custom avatar pictures or even graphs). There are various
> limitations here - the most important one being the problems around resizes,
> but it's the best thing we have.
> One could use cairo_surface_t for this, but I'm not very happy with exposing
> it, because cairo_surface_t has too many weird additional features, like
> device scales, device offsets, it being Cairo backend-bound, its ability to
> be changed at will and last but not least it's just not a GObject.
> What you really want is some sort of object that has width, height and a
> draw function and have lots of subclasses for pixbufs, gradients, single
> color, whatever. But such an object doesn't exist.
> https://git.gnome.org/browse/gtk+/tree/gtk/gtkcssimageprivate.h is what GTK
> uses internally for the different image types CSS supports. You can look for
> the various subclasses that implement those from url() and linear-gradient()
> to -gtk-icontheme() and cross-fade().

gdk_cairo_surface_create_from_pixbuf() already exists. If we moved this
sort of call into gdk-pixbuf, it could know about the internals of the
storage, and convert it more efficiently, without an intermediate step.

> 4. animations
> GdkPixbuf has rudimentary animation API. That API is very sucky from both
> sides - the gif loader is as ugly at implementing it as it is inconvenient
> to use.

What would this API look like ideally?

I think the GIF loader should use giflib instead of its own code, would
reduce the maintenance burden and probably make the code easier to read.

>  Animations are separate from regular pixbufs, so you often need
> multiple code paths if you want to support animations. But it's the only
> thing we have and you don't have to think about it when passing a filename
> to GtkImage, so it's seen various usages.
> There's no good replacement. All solutions that really support animating
> images are really complex. SVG requires full renderers (usually web
> browsers), even rudimentary movies require a GStreamer-like framework and
> simple animations (like displaying progressive image loads) just aren't done.

Right.

> 5. simple drawing operations
> This is the only thing that has been replaced in all code that isn't
> unmaintained or dusty because Cairo is just that neat to use, even if it's a
> bit awkward to first. The only place where it is still used is where there's
> operations that people used to do in 1990 that don't have a modern
> equivalent and people still want to behave like those old apps.

Completely agree here.

> So to reiterate, I wish 3 things would happen:
> - graphic-heavy apps like eog or gthumb would go use Gegl/libgimp. They'd
> get nice filters for free!

Problems being that:
- GEGL can only load from local files (no stream API, no incremental
loading), though it can load GdkPixbufs.
- GEGL uses gdk-pixbuf's loaders

So this hangs on to GdkPixbuf's storage not being very efficient. If we
used babl internally, we could probably short-circuit the conversion
from the internal/native format to GeglBuffer.

> - there would be a nice library for "an area with width and height that can
> draw itself", complete with implementations for drawing with cairo and image
> loaders, icontheme and animation support and hooks that are convenient and
> performant enough to use for GStreamer video implementation and resizable
> animated SVGs.

Which would be something other than gdk-pixbuf.

> - gdk--pixbuf to go away.

But:
- we still want the loaders it provides
- we still need GdkPixbuf, the object

I think that it means that we should still be fixing the loaders if
there are bugs in them, and modifying GdkPixbuf's storage to be more
efficient.

We could certainly nuke the rest if we think that other libraries would
be better equipped for resizing, clipping, scaling, and compositing.