Re: KDE 2.0 impressions

[Alexander Larsson <alla lysator liu se>]

On Tue, 31 Oct 2000, Padraig O'Briain wrote:

> One of the things that we found on Solaris which improved startup performance 
> for applications was the use of scope mapfiles. I have not seen these used in 
> GNOME so perhaps scope mapfiles are not available on Linux. Could someone 
> educate me on this point? One can think of a scope mapfile for a shared library 
> as being similar to a .def file for a Windows DLL in that it defines the 
> interface provided by the library, i.e. the functions in the library which can 
> be acccessed outside the library. The problem with a shared library is that, by 
> default, i.e. without a scope mapfile, every function in the library is exported 
> but in Windows the default is that no functions are exported.
> 
> If using C++, the function names, those mangled ones, tend to be even longer 
> than the function names used in C and the improvement in startup time from using 
> scope mapfiles for the shared libraries for a large C++ application, i.e. 
> StarOffice was significant. The size of the symbol table reduced significantly 
> which reduced the cost of loading it and the cost of finding external references 
> was reduced as the symbol tables were smaller. It is worth noting that external 
> references are to symbol names, without the shared library which contained them 
> at link time being specified. On Windows external references are to a reference 
> number in a specific Windows DLL. This makes fixing up references much faster
> on Windows although if someone changed the order of the functions in the DLL 
> since your application was built you will have some problems.

This was a *huge* problem in mozilla. Due to XPCOM each component dll
really did only export four functions, but by default each bloody non-
static function/method got exported. And due to the c++ name mangling the
dynamic symbol table got *really* large.

The first step towards fixing this was to use a hack with the gnu linker
using ELF symbol versioning. I came up with this hack and ramiro
implemented it in Mozilla.

Basically you create a versioning script like this:
(mozilla/build/unix/gnu-ld-scripts/components-version-script in mozilla
cvs):

EXPORTED {
   global:
                NSGetFactory;
                NSGetModule;
                NSRegisterSelf;
                NSUnregisterSelf;
   local: *;
};

This script makes all symbols except the specified 4 local. As a bonus
these 4 get an ELF version called "EXPORTED", but ignore that.

The .so's are then linked using an extra gcc flag:
 -Wl,--version-script,$(BUILD_TOOLS)/gnu-ld-scripts/components-version-script

When you look at these files with "nm --dynamic" everything looks just
peachy, the unwanted symbols are gone. This is in fact not 100% true. Due
to a bug in the gnu linker (it's marked with "TODO:" in the binutils
code) the extraneous symbol strings in the .dynstr segment are not
removed. This is a bit bad, because it is mostly the strings that are
large, not the symbol table entries.

To fix this i had to write an amazing little utility called elf-gc-dynstr
that removes all unreferenced strings from an ELF shared library, and then
compacts the file. This utility can be found in
mozilla/tools/elf-gc-dynstr in the mozilla cvs tree. Don't look at the
code, your brain will melt.

I've got local modifications to elf-gc-dynstr which makes it work on ppc
to (the current cvs version does x86 only), which i will check in soon.

I don't know if the Gnome project is interested in using this. It was done
mostly to minimize Mozilla for use in embedded systems (desktop systems
gain a lot too of course). 

> Solaris also have a feature which allows the code in shared libraries to be 
> reordered at link time so that frequently accessed code is clustered together 
> and code which is very infrequently used is at the end of the file and is never 
> paged in. Windows also has a similar feature for DLLs. Use of such a feature can 
> reduce the working set size and paging activity for memory constrained systems. 
> This feature was invaluable when trying to get CDE performance to an acceptable 
> level on 32 Mb systems.
This is basically a description of nat's program "grope". Unfortunately he
is to busy hacking gnome to have any time to finish it.
 
> Another feature which is the default on Windows and is available as an option on 
> Solaris and I do not know its state on Solaris is to load shared libraries only 
> when required rather than automatically on startup. This can make a significant 
> difference to the user-perceived startup time if not all shared libraries have 
> to be loaded before the user perceives the application to have started.
What do you mean by "load"? Linux typically mmaps the libraries into
memory, so no page is loaded from disk until it is read from or written
to. Fixup of symbols is lazy by default on linux (each symbol is fixed up
the first time it is referenced), but you can force non-lazy binding by
defining some environment variable (LD_BIND_NOW?).
 
> It is not clear to me whether there is a performance problem on GNOME but if 
> there is, hopefully, we have the tools to fix it.

We have some tools.

/ Alex