Re: Proposal: g_set_out_of_mem_handler

[Sebastian Wilhelmi <wilhelmi ira uka de>]

Hi, again
> >
> > We recently had a discussion on orbit-list about the current fail at
> > "out of memory" behaviour of GLib. While I still think, out of memory
> > can in general not be handled sane, we might as well add a hook to be
> > called whenever Glib find itself out of mem. For the greatest possible
> > generality this routine only have to return a pointer to the requested
> > memory, thus this memory can be taken from a preallocated place in
> > memory, and now comes the clue: gmem.c keeps a list of pointers, that
> > are obtained that way. This list is normally empty (almost no overhead
> > in testing it) and if not empty every freed memory is testet against
> > that list and if contained freed with another callback.
> 
> note that -using this approach- after a short period of memory
> bottleneck, you might end up with a significant amount of small memory
> portions (a bunch of which can actually be RO allocations) that are all
> kept in your pointer list and are checked against upon all further
> free() operations (a very frequently used function btw, due to the
> highly dynamic nature glib/gtk code is coded in general). this can cause
> a major slowdown for the rest of the application's runtime, eventhough
> the real bottleneck quickly vanished.

Quite true, but my argument is: Better slow than failure.
 
> > That way you can preallocate, say, 500 KB and whenever an "out of mem"
> > occurs you just give out the requested memory and be fine. (and hope,
> > XLib will not try allocate memory, that is ;-).
> 
> which is pretty naive as soon as you get into matters of handling the
> event queue...

Yes, in fact. I did various tests and it turned out, that making only g_malloc
have some reserved memory wont work at all. You also have to intercept the
Xlib malloc calls, which simply isn't possible in a 'clean' way. All you can
do is "overloading" the standard 'malloc' and 'free' functions to your liking.
(This might be quite unportable, but it works for electric fence and friends)

> > BTW.: Now, that we have memprof (Great work...), isn't it about time
> > to remove some of that memory profiling in GLib. Its really looking
> > ugly and I doubt, anybody use it...
> 
> ok, i should note first that i wholeheartedly agree with you that
> out-of-memory situations can't be handled comprehensively (and
> portably!) in a sane way at the application level.

Actually seeing my new solution I'm not that convinced any more.

> but since the "broad public" (whatever that is ;) seems to feel a need
> to be enabled to handle this situations at least in edge cases, i think
> we should come up with a more generic solution that will:
> 
> 1) get rid of the old half-backed debugging/"profiling" code in gmem.c.
>    we got three different defines to alter allocation/reporting
>    behaviour
>    currently, which tend to be rarely used and over-complexify glib's
>    basic allocation strategies.

Yes. Seconded.

> 2) be comprehensible enough to cover future feature requests that may
>    popup
>    in this area, for instance to facilitate platform specific allocation
>    debuggers or memory-region locks (the interested linux user may want
>    to
>    ponder about mprotect(2) for a moment).
> 
> some considerations that imho should be taken into account:
> - we certainly can't feature something as versatile as the GAllocator
>   mechanism, since g_malloc() is used for probably 90% of a normal gtk
>   program's allocations at all programming layers, including the 
>   low-levelglib stuff.
> - this seems like a good opportunity to make an effort on specializing
>   for read-only allocated regions, so we
>   * avoid a major portion of so-thought-leak "bug reports"
>   * can use provided hooks to report real leaks and meaningfull
>     statistics of the g_malloc() allocations being made by the
>     application
>   * provde hints for the fallback-allocation routines used upon EMEM, so
>     it can deal with allocation requests cleverly, i.e. not bother about
>     RO regions being freed.
>   (i estimate about a 1/2 to 1 day (for a core hacker) to go over
>   glib/gtk code and substitute ro allocations with the new function
>   calls)

Ok, I really dont see much sense in those RO allocations apart from better
memory leak reporting, So I'm not sure, we want that

> i think a good aproach on this, would be:
> 
> typedef struct _GMallocTable GMallocTable;
> struct _GMallocTable
> {
>   gpointer (*malloc)            (gulong         size); /* obligatory */
>   gpointer (*romalloc)          (gulong         size);
>   void     (*free)              (gpointer       mem);  /* obligatory */
>   gpointer (*fallback_malloc)   (gulong         size,
>                                  gboolean       ro_allocation);
>   void     (*report)            (const gchar   *log_domain,
>                                  GLogLevelFlags log_level);
> };

Ok, but how do we handle realloc? Even if we has a 'gpointer (*realloc)' in
the above structure, we couldn't use it, because what should we do if somebody
calls g_realloc and table->realloc would return 0, we had to
table->fallback_malloc the region and copy the data ourself, but how many
bytes? right MIN(old #bytes, new #bytes), but we don't know old #bytes, so
we're hosed with a potential segfault, if we copy to much bytes. Even
table->realloc_retry wouldn't help, as that functions doesn't know more than
we do. Solution: Save the length of each allocation in front of it. I mean:
that can't be worth it.... So we have a problem. 

The other alternative to provide a callback for the "out of mem" case is just
as stupid, because we simply can't do anything usefull in that callback. All
kinds of mutexes might be locked, almost every piece of code in GLib assumes
not to be called recursivly, when calling g_malloc. So we can't free e.g.
mem_chunks in such a callbeck, let alone open a window telling something went
wrong, thus rendering such a callback very useless.

So, what did I do then:
 
I simply added a new lib to GLib (which is something like gthread and
gmodule): gmemres (g memory reservation).

It contains a (actually only slightly) hacked version of Doug Lea's malloc, as
found in glibc[1]. This simply makes sure, that between the last allocation
and the real end of the heap, as allocated from the OS with sbrk, there will
always be g_reserved_mem_size (set it with g_reserved_mem_set, which will
return TRUE, if the reservation succeded) bytes left. If that wasn't possible,
then the function 
g_reserved_mem_used will return TRUE. Then I provide the function 

typedef gboolean  (*GOutOfMemCallback) (void);
guint    g_out_of_mem_add_watch (GOutOfMemCallback callback);

that makes the callback be called from the main loop, whenever that reserved
mem is used. There we can do many things, as long, as we don't use more memory
than reserved. 

I hacked together an example, that is appended (mostly taken from testgtk, but
shows, what can be done.)

So my opinion is: Either we do it in a way similiar to mine or we just let
this topic die for GLib.

[*]: Actually in glibc there is a thread optimized version from Wolfram Gloger
<wmglo@dent.med.uni-muenchen.de>. But I looked at it and my hack would work
there too, it would only be slighly more challenging to make it thread safe
for GLib. While the current code (in my Glib extension) is not yet thread safe
either it is quite simple to make it so.

So whoever want, take a look at my example:

first download the patched glib-version and the example:
	
	ftp://i81pc4.ira.uka.de/glib-memres.tar.gz
	ftp://i81pc4.ira.uka.de/glib-memres-example.tar.gz

then configure, compile, install it somewhere different from the standard
place. Adjust the Makefile of the example accordingly (GLIB_PATH) and type
'make run' and add some 1000's of lines. (Not the 10000 lines though, the
reserved mem is not big enough for them, this demo is just a hack ;-) 

Bye,
Sebastian

-- 
Sebastian Wilhelmi                   |            här ovanför alla molnen
mailto:wilhelmi@ira.uka.de           |     är himmlen så förunderligt blå
http://goethe.ira.uka.de/~wilhelmi   |