Re: An alternative to gdk-pixbuf

[Magnus Bergman <magnus bergman snisurset net>]

On Thu, 06 Sep 2018 13:03:03 -0500
Federico Mena Quintero <federico gnome org> wrote:

> On Wed, 2018-09-05 at 17:28 +0100, Emmanuele Bassi via gtk-devel-list
> wrote:
>
> > In the near future, I'll very likely deprecate most of GdkPixbuf's
> > API, except for the I/O operations; I'd also be happy to seal off
> > most of its internals, within the ABI stability promise, to avoid
> > leakage of internal state.  
>
> Related to this - I just wrote a braindump on gdk-pixbuf issues and
> historical baggage.  I hope it's useful for people trying to clean up
> this swamp:
>
> https://people.gnome.org/~federico/blog/my-gdk-pixbuf-braindump.html

This was an interesting read. It touched many of my own ideas and
concerns.

Animation

I actually think the concept of animations in gdk-pixbuf is okay, so
it's pretty much the same in abydos, with the main difference that
abydos fully supports vector animation. Apart from the frame based view
of animations abydos also support continuous animations (which it seems
like gdk-pixbuf also does). It's mostly the API of gdk-pixbuf that is
suboptimal. Apart from GIF there are a few formats common in the art
scene that supports animation. It's not just flip book animations but
also palette cycling or simply one blinking color. There are also
formats that show the current time (or depend on other variables that
can change over time). Ideally an image viewer should update those
then appropriate, so they can also be considered animations. (As you
might have noticed I'm quite interested in the odd corner cases.)

Mutable pixbufs

The problem with mutable pixbufs is pretty much the same problem
described in the documentation for abydos_get_image_surface(). The user
can promise to either not modify pixels or not use the image for
anything else after the modification (this is of course the fastest
alternative and currently the default). Or the user can request the
pixels to be copied or transferred, in which case it's safe to both
modify the pixels and use the image for something else. This solution
is a bit awkward, but since I think the copy/transfer scenarios are
actually corner cases (which I still want to support) the main use case
should remain as fast as possible, and also the one not requiring
extra typing.

Pluggable loaders

Pluggable loaders it's something I very much appreciate since I can
write my own loaders for the additional formats I care about (ten or
so) and use common image viewers to view them. Otherwise I would need to
write the whole viewer myself as well. The only alternatives are image
viewers on original hardware (emulators may or may not cut it since the
image viewers often use hardware tricks that are difficult to emulate
and most of them doesn't even try very hard to correct the aspect
ratio). Or try to get some strange and glitchy Windows application
running in Wine, often with a suboptimal result since it doesn't
support all details I care about (such as a correct aspect ratio and
palette cycling). Some art groups release their own viewers, that are
usually quite good. But they usually only support the subset of
features their own paint applications support. One viewer to view them
all would be better. But all your favourite viewers to view them all
would be the best IMHO.

Multi page files

Multi page TIFF images and SVG images with multiple icons inside are
conceptually the same to abydos. The same API could support both.

Limitation of supported image formats

It's clear that GTK+ doesn't need to load that many image formats. Most
applications doesn't either. And it's clear that more supported formats
is a bigger security risk. (But I would like to add that this isn't
limited to image loading plugins, in general more software is less
secure than less software). For image viewers I think the priorities
are different. Most would opt for more more more. (It's not like people
would opt for less games on steam just because that would reduce the
attack surface, to draw a parallel.) Also, distros would probably
package the more exotic loaders in different packages, not installed by
default.

In case applications shouldn't be allowed to read any kind of image, a
filtering could be installed that only let blessed MIME types through.
It could easily be added to a wrapper around abydos. Apart from the
security aspect I think it's a good thing if an application can open
anything you throw at it. In the word processor example you mentioned
it's especially true IMHO. Personally I use Lyx, and it gives you the
ability to teach it about new formats if you just tell it how to
convert it to a format it already knows about. I use that a lot and
find it very useful and time saving. But it's a problem if you expect
others to be able to open it (which I don't, I export it to PDF
instead). So for any kind of format intended for data exchange it
should be limited, and clearly defined, what you can embed in it.

In the end it seams to boil down to this. You can only pick two:
  1) comprehensive image format support.
  2) security.
  3) speed.
I think everybody could be happy if the choice could be made at
run-time.

Progressive loading

I also doubt there are some actual use case that needs progressive
loading. A progress bar would probably do in the cases it takes a
little time to load an image I think. In abydos that API is always
supported, but it's not required for plugins to support it. In case
they don't abydos will buffer the bytes, call plugin with all data
available, then invoke the callbacks to the user supplied functions.
I'm quite sure it wouldn't break anything if gdk-pixbuf did the same.
It could be mentioned that some image formats cannot be progressively
loaded anyway since crucial data is placed at the very end (or they
have no predefined height, just add rows until there is no more data).

The feature to request another size than the reported size was mainly
added to support vector images at larger scale I think. I'm even quite
sure it wasn't intended for taking a short path then a smaller image
was requested. Once I suggested that if an image contained an embedded
thumbnail, that could be returned if the user requested that size (or
smaller), especially then creating thumbnails. I was told no, images
should always be loaded at their largest size and then scaled down.

By the way. Back 1998 I don't think loading images over a slow internet
connection was the only use case for progressive loading. You could
definitely see images load progressively even then loading from disk too
(and I'm not even talking about floppy disks).

Non-raster image formats

For non-raster data abydos solves the problem by being agnostic to what
an image can be. It's simply anything that can be rendered using cairo.
This made it especially easy to make plugins for SVG and PDF since
librsvg and poppler has this exact same approach.

Meta data

Then it comes to meta data abydos is almost useless, it only cares
about meta data that affects viewing the image. Trying do do more will
either be overly ambitious or half assed. I think there is no middle
ground. For general purpose meta data extraction libextractor is most
useful (but not perfect). And I don't think this is something GTK+ needs
at all. For loading one format and saving to another users might expect
the meta data to be preserved to some degree, but to what degree is
completely arbitrary and I guess it's impossible to find one that makes
everybody happy. I think it's better do to what netpbm does, simply
ditch all meta data. But I'm glad you mentioned color space support.
That's something I have to think about.

MIME sniffing

I was pretty sure the custom MIME sniffing was already removed from
gdk-pixbuf (according to another answer it seems to be a compile time
option). Otherwise it's done twice. Because then I write a loader I need
to add the corresponding MIME type to the shared MIME database as well
for it to work. In case it has been removed, I think the pattern field
(as well as the extension field) in all loaders should be left empty. (I
assume the ABI isn't intended to be stable in that direction.)

Rust

I also like the idea of writing plugins in Rust. My experience with
Rust is very limited (and a bit dated). But to me Rust seams to be best
suited for large applications. It's quite possible to create very tiny
binaries, but as soon as you use one something from a library tons of
stuff gets dragged in (as with any language). And it gets very
noticeable since everything is statically linked. It seams you have
better experience with Rust and I look forward to see what you come up
with.

The main question, at least for me, is: Should the image loading
library that GTK+ uses to load images be a general purpose image loading
library also usable for image viewing applications?

It seams clear to me that the general opinion points to no. And that
the goals of abydos doesn't match the needs for GTK+. The problem
with the current situation is that creators of image viewing
applications perceive gdk-pixbuf as a general purpose image loading
library and uses it for that purpose. One could say that the interest
of GTK+ developers are in conflict with the interest of users of image
viewing applications. In that case the obvious solution is to clearly
state that gdk-pixbuf is not a general purpose image loading library,
but only intended for loading icons for GUI elements. (And possibly let
the whole thing die in time for GTK+ 5.)

> Magnus - thanks for showing us Abydos.  The parts of the API that
> intersect with gdk-pixbuf's look okay.  I think you may find this
> useful:
>
> https://developer.gnome.org/programming-guidelines/stable/index.html.en
>
> I think your API could improve with some of the material there around
> memory management and error reporting.  Internally, the code needs
> checks in all the malloc()-like functions; some of the arithmetic
> (especially in assertions) needs to avoid overflows.  I couldn't find
> any tests.  The SVG plug-in needs to handle error results from
> librsvg. The Magick plug-in for PNG and JPEG has no error handling;
> also you may want to call cairo_surface_mark_dirty() when you are
> done writing pixels to it.  The netpbm plugin does a bunch of system
> calls and doesn't check for errors.

Yes, the code is a bit sketchy right now. I'm still in a phase then I
consider throwing away and rewriting large parts. I will definitely add
better error handling before 1.0 (and well before that hopefully). I'm
mainly focusing on getting the API right at the moment. But I appreciate
you point these things out (I'll write them down so I don't forget
about them).

There are tests (no comprehensive test suit though), but they all rely
on non-free images, so I decided to leave that out for now.

This is a bit off topic. But are there actually modern (non real time)
systems that can recover from a state there malloc() has failed?
At least in Linux it will basically only happen if you run out of
memory addresses (which will not happen on a 64-bit system at least).
And if that happens you're long since screwed anyway. Since the actual
allocation will happen then you write to the memory, malloc() will
continue to give you memory that doesn't exist. I just assumed any
modern non real time system takes this approach. (And if one aims for
true real time compatibility it opens up a whole new can of worms.)

> The parts of your API that deal with pagination / variants / animation
> frames - I'm not sure if a gdk-pixbuf replacement would need them, but
> I'm coming from the viewpoint of my braindump above.

It's not pagination, as in chopping up an image for printing it on
several pages. The concept of pages (in lack of a better term I guess)
is about files containing multiple images with an otherwise unspecified
relationship. Such as SVG files containing multiple icons. The API is
also designed so that everything related to pages, layers, variants and
animations can just be ignored, both by applications and plugins, then
not needed. (And the extra bytes it adds to the binary is not
overwhelming.)

Anyway, thank you very much for your valuable feedback!