Can't launch apps, fork() returns ENOMEM

[Daniel Drake <drake endlessm com>]

Hi,

At Endless we're looking at how to make GNOME run better on systems
with low amounts of RAM.

One issue biting on a regular basis particularly on systems with (say)
1GB RAM is that apps will refuse to launch from gnome-shell even when
there's a decent enough chunk of memory available.

The user sees error message:
   fork(): Cannot allocate memory

When gnome-shell is launching apps (via glib) it ultimately comes down
to fork() + exec(). In this case the fork() fails with ENOMEM, because
the Linux kernel worries that the process being forked may end up
duplicating all of the memory allocations of the shell. By default the
memory map is set up so that the new process has a view on the exact
same pages as the parent process, however they are set up as
copy-on-write, so if the child writes to such memory it'll silently
cause new memory to be allocated. Under that limited perspective, the
kernel is not totally out of line in worrying about this situation,
especially because gnome-shell is a RAM-heavy process.

In reality we only want to fork() to immediately exec() which replaces
the child process memory map with a blank slate, but this
misinterpretation of intentions is a limitation of the fork() API
combined with a conflict with Linux's memory overcommit model.

The solution to this leads us towards alternatives to fork(). vfork()
can create the forked process using the exact same memory address
space as the parent process, meaning that there's no memory that might
need to be duplicated, avoiding this failure condition.
clone(CLONE_VM) can do the same thing. But two processes using the
exact same memory draws in a whole bunch of nasty complications, it's
dangerous.
https://ewontfix.com/7/
https://gist.github.com/nicowilliams/a8a07b0fc75df05f684c23c18d7db234

Discussions here have lead to glibc's posix_spawn() being recently
reimplemented to use clone(CLONE_VM) while solving practically all the
danger there. How can we do something similar in GNOME?

The complications mostly boil down to that you need to be rather
careful in what you do in the child process before you exec(), since
you are sharing the parent's memory. And the underlying
g_spawn_async_with_pipes() is a powerful interface which can do a
number of things in that danger zone, depending on flags passed in.

I started to implement an alternative codepath for
g_spawn_async_with_pipes() to execute when the flags aren't
particularly complex (which appeared to be the case for app
launching), based on the posix_spawn() design. But I ran into too many
headaches and brick walls there. For example you have to allocate a
stack for the child process so you need to know if the stack grows up
or down on your architecture. glibc knows these architectural details
well, but we don't. Then the games with signal handling - ideally we
need to block glibc-internal signals (e.g. NPTL stuff) in the child
process, but that's also a hairy task to do outside of the glibc
codebase. etc.

Then I considered making glib just call posix_spawn() directly when
appropriate. There are a number of things in that API which let
relevant actions be done in the child process before it does the exec
- managing file descriptors, signal handlers, etc. The big issue there
is that glib lets you pass an arbitrary GSpawnChildSetupFunc to be run
in the child before the exec(), this is used by gnome-shell, and there
is no equivalent hook in posix_spawn().

In gnome-shell app_child_setup() this function is used to send stdout
and stderr to journald. Also in glib gdesktopappinfo.c child_setup()
the same mechanism is used to set GIO_LAUNCHED_DESKTOP_FILE_PID in the
child process to it's own process PID (can't find the background
here).

I think there are 2 viable paths forward:

 1. Eliminate the child_setup calls from these codepaths to allow
posix_spawn() to be used. The gspawn code already allows for file
descriptor redirection but this would have to be exposed via
additional parameters to g_desktop_app_info_launch_uris_as_manager() -
an API change. If GIO_LAUNCHED_DESKTOP_FILE_PID can't be reimplemented
another way then we could first exec a binary wrapper that sets this
env variable before execing the app itself.

or

 2. Propose to http://www.opengroup.org/austin/ that posix_spawn()
grows the capability to call a user-specified child setup func. There
is some unused padding in posix_spawnattr_t which could be used to
store the function pointer and data pointer. Then implement this in
glibc. I guess this is a lengthy process and it's not a great idea
from the glibc standpoint where such code would run in dangerous
context.

I'm inclined to persue the first approach, of removing child_setup
from this codepath and then implementing an optimized gspawn codepath
that uses posix_spawn() when the conditions are right.

I'd appreciate any comments before I continue though, does this design
sound acceptable? Am I missing anything?

Thanks,
Daniel