Re: An alternative to gdk-pixbuf

[Debarshi Ray <rishi is lostca se>]

Hey Magnus,

I haven't yet worked my way through the whole thread. It's pretty
long and will take me a while longer, but I did want to mention a
few things before the weekend draws me away from the computer.

On Wed, Sep 05, 2018 at 12:02:45AM +0200, Magnus Bergman wrote:
> Over the years it has been discussed from time to time to replace
> gdk-pixbuf with something else[1][2]. Something was even in the making
> (I guess over ten years ago) but it never replaced gdk-pixbuf
> apparently. Now I don't even remember what it was called. And something
> else called pig was proposed more recently.

As Emmanuele mentioned, if you ignore the use-case of loading icons
in GTK, and focus on image applications dealing with high resolution
images, then GEGL [1] is the way to go. It's a GObject-based image
processing framework that's used by GIMP and GNOME Photos, and is
way more advanced than anything that we have in the GNOME platform.

If you use GIMP 2.10 or practically any version of GNOME Photos, then
all the image decoding and processing is happening with GEGL.

A GeglBuffer is the equivalent of GdkPixbuf in GEGL. It can:

(a) Be converted to and from a GdkPixbuf. That makes porting trivial
    and boosts interoperability.

(b) Handle massive images that are larger than the amount of physical
    RAM available on the system. This is achieved by spliting the
    pixels into smaller tiles that get transparently swapped out of
    memory into a disk-backed cache when necessary.

    It can be dumbed down into a GdkPixbuf-like linear sequence of
    bytes, if need be.

(c) Represent a whole horde of pixel formats, colour spaces, bit depths,
    etc. [2], which can be programmatically extended at run-time.

(d) Automatically convert between pixel formats, colour spaces, bit
    depths, etc., with accelerated fast paths.

(e) Be serialized into a named file, which is interesting for passing
    pixels across process boundaries.

(f) Use mipmaps for scaling, rendering and processing the pixels.

(g) Be used with OpenCL.

(h) Do all that and is still fast. Try zooming in and out in GNOME
    Photos.

Not to mention a vast array of image processing operations. If it's in
GIMP, it's in GEGL. Interesting operations are ported from elsewhere,
and there's a constant flow of new things under development.

> One major reason to replace gdk-pixbuf has been the long term goal to
> phase out GDK in favour of cairo.

De/encoding images and manipulating the pixels is related to rendering
them on the screen, but aren't necessarily the same thing. It's
important to ensure that the pixels can be efficiently drawn, but I
wouldn't add a hard dependency on the drawing framework. For example,
GEGL has fast paths for drawing with Cairo, but doesn't mandate it.
It also doesn't require GTK, even though it has helper widgets for it.

That's why GIMP 2.10 and GNOME Photos can use the same libgegl-0.4.so,
even though they use different widget toolkits (GTK 2.x versus 3.x).

> This is how to convert an SVG file to a PNG file.
>
>     cairo_surface_t *surface = abydos_load("image/svg+xml",
>     "example.svg"); cairo_surface_write_to_png(surface, "example.png");
>     cairo_surface_destroy(surface);
>
> This is how to convert the second frame of a GIF animation to PNG.
>
>    abydos_t *ar = abydos_create_from_file("image/gif", "example.gif");
>    abydos_set_frame(ar, 1);
>    cairo_surface_t *surface = abydos_get_image_surface(ar, 0);
>    abydos_destroy(ar);
>    cairo_surface_write_to_png(surface, "example.png");
>    cairo_surface_destroy(surface);

I don't see any error handling. Maybe I should read the actual code.

Either way, as Emmanuele mentioned, it's really important to have
support for GIO facilities like GCancellable, GFile, streams,
GAsyncResult-based asynchronous APIs, etc..

If you look closely at GEGL, you'll see that while it already has
codecs, they are missing these requirements. It's an artifact of
GIMP historically having it's own in-house codecs.

However, that's changing. I have a work-in-progress GNOME Photos
branch [3] that prototypes a better codec API for GeglBuffer.

See:
* https://gitlab.gnome.org/GNOME/gnome-photos/blob/wip/rishi/buffer-decoder/src/photos-gegl-buffer-io.h
* https://gitlab.gnome.org/GNOME/gnome-photos/blob/wip/rishi/buffer-decoder/src/photos-gegl-buffer-loader.h

One obvious thing is that it looks almost identical to the corresponding
GdkPixbuf API. That's an explicit goal to make porting trivial.

Of course, one could use the GdkPixbuf codecs for I/O and then covert
to and from a GeglBuffer. But then you wouldn't be able to handle RAWs
or higher bit-depth PNGs and will likely run into problems with your
80 megapixel images [4] - things that a good image application will
care about. :)

> I've been in contact with a two creators of image viewers

Which ones? Just curious. :)

> What it all comes down to is that using a bigger variety of software
> creates a bigger attack surface. Viewing a lot of images is less secure
>
> [...]
>
> formats (MIME types) in specially vulnerable situations (like for auto
> displaying images from internet). And for extra security all but a core
> set of loaders could be removed.

Flatpak plays a big role here. If the application is sandboxed, the
attack surface decreases quite a bit.

As Ray said, you can also move the decoding into a separate process
that's even more tightly locked down than the main application. This
is what GIMP does. You can serialize the GeglBuffer to transfer the
pixels, without using any OS-specific facility, which is a nice touch
if you want to be portable across Linux, Windows, OS X, etc..

Happy hacking,
Rishi

[1] http://gegl.org/
[2] http://gegl.org/babl/
[3] https://gitlab.gnome.org/GNOME/gnome-photos/tree/wip/rishi/buffer-decoder
[4] https://en.wikipedia.org/wiki/Phase_One_(company)