[RFC] linker magic

[muppet <scott asofyet org>]

Gtk2-Perl currently relies on DynaLoader's dl_load_flags() method and
the RTDL_GLOBAL flag.  The attached patch attempts to make this
obsolete.

I would appreciate questions and comments from anybody with ideas about
maintenance and linkage and such guru topics, so please excuse the
cross-post to language-bindings.  It's a blatant attempt to boost the
chances of hitting such knowledgeable people who have probably solved
this problem for their own bindings.


Most Perl extensions (Perl code that calls out to C code) consist of one
Perl module and one shared object.  The module bootstraps the shared
object; the shared object calls symbols in the main perl executable at
runtime, and exports symbols which are set up to be callable from the
interpreter.  These extensions are usually self-contained.

Gtk2-Perl is a little different from usual extensions in that the
extensions depend on one another in terms of C symbols; the Glib module
provides helper functions such as gperl_get_object() and
gperl_value_from_sv(), and other important bits of code which are used
extensively by the other bindings (Gtk2, Gnome2::Canvas, Gnome2::VFS,
etc, etc).

Gtk-Perl (the bindings for gnome 1.x) also had to solve this problem;
their solution was to make use of the runtime dynamic loader's ability
to make all of the symbols in the freshly-loaded object globally
available, that is, visible to all other portions of the application. 
Gtk2-Perl inherited this behavior.

However, as we take gtk2-perl to different platforms, this has proven
problematic.  RTDL_GLOBAL is not supported on all platforms, and in fact
has to be implemented with a magic number (0x01) in a method named
dl_load_flags() (in other words, it's an ugly hack); Darwin complains
about the flag, but leaves the symbols available anyway; HP-UX is
reported not to handle it at all; and Win32...  Well, Win32 requires
that all symbols be resolved at *link* time, so we actually have to dig
through the perl library to find the dependent extension dlls and add
them manually to the link commands, rendering RTDL_GLOBAL irrelevant.


Which begs the question: if we already have to find the objects for
explicit linking for win32, why not just do that on all platforms and
stop fooling with the non-portable RTDL_GLOBAL/dl_load_flags nonsense?


We have ExtUtils::Depends, a utility module that handles most of the
grunt-work of creating the parameters to pass to ExtUtils::MakeMaker to
create the Makefiles for the extensions.  Depends already knows what
modules we need, and collates linker flags, so it's pretty simple just
to trawl the library for the shared objects and add them to the list of
objects at link time.  The attached patch does this, and removes the
dl_load_flags() from Glib, Gtk2, and Gnome2::Canvas just to prove that
Gnome2 (which depends on C symbols from all three of those) can still
link and run.

This stuff works fine on Linux, but i have a few concerns:

1) will it work on other platforms?  (if somebody could try this patch
on both native win32 and cygwin i'd really appreciate it.)

2) what kinds of pitfalls are waiting down this path?  (e.g. implicit
paths, symbol clashes, etc)

3) will this break binary compatibility?  (i think the safe solution is
to leave the dl_load_flags() hacks in place for compatibility with older
binaries, otherwise they will need to be recompiled)


Index: ExtUtils-Depends/lib/ExtUtils/Depends.pm
===================================================================
RCS file: /cvsroot/gtk2-perl/gtk2-perl-xs/ExtUtils-Depends/lib/ExtUtils/Depends.pm,v
retrieving revision 1.11
diff -u -r1.11 Depends.pm
--- ExtUtils-Depends/lib/ExtUtils/Depends.pm	12 Mar 2004 21:02:22 -0000	1.11
+++ ExtUtils-Depends/lib/ExtUtils/Depends.pm	22 Jun 2004 03:19:19 -0000
@@ -282,8 +282,10 @@
 	my %vars = (
 		INC => join (' ', uniquify @incbits),
 		LIBS => join (' ', uniquify @libsbits),
+		#LIBS => join (' ', uniquify $self->find_extra_libs, @libsbits),
 		TYPEMAPS => [ typemaps],
 	);
+	push @OBJECT, $self->find_extra_libs;
 	# we don't want to provide these if there is no data in them;
 	# that way, the caller can still get default behavior out of
 	# MakeMaker when INC, LIBS and TYPEMAPS are all that are required.
@@ -299,6 +301,37 @@
 	%vars;
 }
 
+# $Config{so} = 'so' on linux and 'dll' on win32
+# $Config{lib_ext} = 'a' on both.
+sub find_extra_libs {
+	my $self = shift;
+	use File::Find;
+	use Config;
+	my @ret;
+	my $ext = $Config{so};
+	# XXX what about linkage order?  these should be in the most-dependent
+	# first order, for old-style linkers...  but when we get the dep names
+	# from the hash, we have no idea about order.
+	foreach my $name (keys %{ $self->{deps} }) {
+		my $dep = $self->{deps}{$name};
+		(my $dllname = $name) =~ s/^.*:://;
+		my $match = qr/$dllname\.$ext$/;
+		my $l;
+		find (sub {
+			$l = $File::Find::name
+				if (not $l) && /$match/;
+		}, @INC);
+		if ($l && -f $l) {
+			warn "found $l\n";
+			push @ret, $l;
+			next;
+		}
+		warn "can't find extra lib for $name\n";
+	}
+	#print Dumper(\ ret);
+	return @ret;
+}
+
 1;
 
 __END__
Index: Glib/Glib.pm
===================================================================
RCS file: /cvsroot/gtk2-perl/gtk2-perl-xs/Glib/Glib.pm,v
retrieving revision 1.66
diff -u -r1.66 Glib.pm
--- Glib/Glib.pm	1 Jun 2004 15:40:30 -0000	1.66
+++ Glib/Glib.pm	22 Jun 2004 03:19:23 -0000
@@ -63,7 +63,7 @@
 
 our $VERSION = '1.050';
 
-sub dl_load_flags { $^O eq 'darwin' ? 0x00 : 0x01 }
+#sub dl_load_flags { $^O eq 'darwin' ? 0x00 : 0x01 }
 
 bootstrap Glib $VERSION;
 
Index: GnomeCanvas/Canvas.pm
===================================================================
RCS file: /cvsroot/gtk2-perl/gtk2-perl-xs/GnomeCanvas/Canvas.pm,v
retrieving revision 1.20
diff -u -r1.20 Canvas.pm
--- GnomeCanvas/Canvas.pm	6 Jun 2004 15:53:13 -0000	1.20
+++ GnomeCanvas/Canvas.pm	22 Jun 2004 03:19:24 -0000
@@ -21,7 +21,7 @@
 	$class->VERSION (@_);
 }
 
-sub dl_load_flags { 0x01 }
+#sub dl_load_flags { 0x01 }
 
 bootstrap Gnome2::Canvas $VERSION;
 
Index: Gtk2/Gtk2.pm
===================================================================
RCS file: /cvsroot/gtk2-perl/gtk2-perl-xs/Gtk2/Gtk2.pm,v
retrieving revision 1.64
diff -u -r1.64 Gtk2.pm
--- Gtk2/Gtk2.pm	10 Jun 2004 22:51:20 -0000	1.64
+++ Gtk2/Gtk2.pm	22 Jun 2004 03:19:24 -0000
@@ -70,7 +70,7 @@
 # complains "Can't make loaded symbols global on this platform" when this
 # is set to 0x01, but goes on to work fine.  returning 0 here avoids the
 # warning and doesn't appear to break anything.
-sub dl_load_flags { $^O eq 'darwin' ? 0x00 : 0x01 }
+###sub dl_load_flags { $^O eq 'darwin' ? 0x00 : 0x01 }
 
 # now load the XS code.
 Gtk2->bootstrap ($VERSION);