[PD-dev] how to load shared code for libraries
Hans-Christoph Steiner
hans at at.or.at
Mon Sep 26 23:29:02 CEST 2011
On Sep 26, 2011, at 2:42 PM, IOhannes m zmölnig wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 09/26/2011 06:52 PM, Hans-Christoph Steiner wrote:
>>
>> IOhannes and I were discussing how to load shared code for a libdir
>> on
>> IRC. The sticking point is that Mac OS X seems to require dylibs to
>> have a hard-coded path. IOhannes posted a test lib to try to work
>> out
>> how to do it using DYLD_LIBRARY_PATH:
>>
>> http://iem.at/~zmoelnig/OSX/test.tgz
>>
>> So here are my results from Mac OS X 10.5.8:
>>
>> creating a [test] object:
>> -------------------------
>>
>> /private/tmp/test/tmp/test.pd_darwin:
>> dlopen(/private/tmp/test/tmp/test.pd_darwin, 10): no suitable image
>> found. Did find:
>> /private/tmp/test/tmp/test.pd_darwin: unknown required load command
>> 0x80000022
>> test
>> ... couldn't create
>>
>>
>> hans at dhcp-10 tmp $ ~/code/pd-extended.git/src/pd -noprefs -lib test
>> -------------------------------------------------------------------
>>
>> ./test.pd_darwin: dlopen(./test.pd_darwin, 10): no suitable image
>> found. Did find:
>> ./test.pd_darwin: unknown required load command 0x80000022
>> test: can't load library
>>
>
>
> just to clarify a bit:
> the binaries posted under the link above were compiled under OSX-10.6,
> without any precautions for backward compatibility.
> they don't work on 10.5, nor 10.4
>
> i also put the same set of binaries online, but this time compiled
> under
> 10.4, and they work and function as expected under 10.4 and 10.6
> (and i
> expect them to work on 10.5 as well)
>
>
>
> on w32, the search paths for dll's should be modifiable via the %PATH%
> environment, whereas on linux (and most i think BSD and hurd) the
> envvar
> in question is LD_LIBRARY_PATH
>
>
>
>
> after the technical details, let's step back and have a look at what
> this is all about:
> the main motivation is to be able to bundle shared code and 3rd party
> libraries with externals.
>
> the dynamic linkers of the various OSs, will automatically search for
> dylinked libraries your external depends on, in various places (e.g.
> /usr/lib).
>
> this is great if you have an easy way to install the needed library
> into
> this place (e.g. a dependy-informed installer system like debian's
> "apt").
> it's not so great, if you don't have such a system, and the user is
> expected to find and download the given dependencies and put them into
> your system's search path.
> even if an external ships with all dependencies, they still have to be
> installed in the "correct" places, which makes installation more
> complicated than just dragging a folder onto your desktop.
> why?
>
> because your OS's dynamic linker doesn't look for 3rd party
> libraries in
> weird places like a folder on your desktop.
>
> now the idea, hans and me came up with, was to make Pd tell the OS to
> look into those places.
> e.g. if library "foo.pd_darwin" gets installed into ~/Library/Pd/foo/,
> then it would be nice, if the dynamic linker would search for any
> depending libraries in the ~/Library/Pd/foo/ directory.
> likewise "bar.pd_linux" (installed in ~/my-externals/pub/) might have
> it's dependencies searched for in ~/my-externals/pub/.
>
> if both "foo" and "bar" depend on "libdudelsack", and both provide a
> "libdudelsack.dll" in their homedirectories, then of course we want
> both
> libraries to choose "their" correct one.
>
> so the idea is, to modify the automatic search path of the dynamic
> linker whenever a library is loaded.
>
> example:
> the user installs the "frotzel" library, by downloading it, and
> unzipping it into /usr/local/pd-externals/
>
> now, if the user requests a new object [frotzel], Pd will search for a
> matching binary and eventually find frotzel.l_ia64 in
> /usr/local/pd-externals/frotzel/
> before dlopen()ing this binary, it will read the LD_LIBRARY_PATH
> variable, push it to some stack, and then add
> /usr/local/pd-externals/frotzel/ at the beginning of the
> LD_LIBRARY_PATH.
> it will then dlopen() the frotzel.l_ia64 binary, and the linker
> discovers, that it has to load the libknurbel.so library, which it
> will
> (magically) find in /usr/local/pd-externals/frotzel/
> because all dependencies are fullfilled, [frotzel] will successfully
> be
> loaded and will work.
> after fortzel.l_ia64 has been successfully loaded, we no longer need
> to
> search for libraries in /usr/local/pd-externals/frotzel/, and
> therefore
> the original content of LD_LIBRARY_PATH is restored (popped from the
> stack).
I got this building and working on Mac OS X, since that's where the
discussion started. It was really just a matter of getting the build
flags right. I am using tkwidgets as my test bed, then I'll integrate
the changes into the library template. You can see my work here in
the 'newentry' branch:
https://github.com/pd-projects/tkwidgets
> the exact name of "LD_LIBRARY_PATH" is system dependent, but it seems
> that LD_LIBRARY_PATH, DYLD_LIBRARY_PATH and PATH cover most systems.
>
>
> fasdmrt
> IOhannes
My guess is that it'll be something like this:
#ifdef _WIN32
PATH
#elif defined(__APPLE__)
DYLD_LIBRARY_PATH
#else
LD_LIBRARY_PATH
#endif
And the IRC transcript for the record:
i don't exactly know how to do LD_LIBRARY_PATH on w32 yet
_hc
IRC
1:54:17
IOhannes: the annoyances don't end there...
1:54:30
if libtest10.dylib links to something esle....
IOhannes
IRC
1:54:31
i _thought_ it's a matter of %PATH% or so, but i never checked
_hc
IRC
1:54:52
there seems to be some vars for dyld to handle this:
IOhannes
IRC
1:54:58
i'm pretty sure there is aa mechanism on w32 that allows us to do that
_hc
IRC
1:54:59
DYLD_FALLBACK_LIBRARY_PATH
This is a colon separated list of directories
that contain
libraries. It is used as the default location for
libraries not
found in their install path. By default, it
is set to
$(HOME)/lib:/usr/local/lib:/lib:/usr/lib.
DYLD_ROOT_PATH
This is a colon separated list of directories. The
dynamic linker
will prepend each of this directory paths to every
image access
until a file is found.
IOhannes
IRC
1:55:20
would you mind posting to pastie?
1:55:37
it's hard to read on irssie
_hc
IRC
1:55:41
http://developer.apple.com/library/mac/#documentation/Darwin/Reference/
ManPages/man1/dyld.1.html
IOhannes
IRC
1:55:46
irsee, that is
1:56:11
excalibas left the room (quit: Remote host closed the connection).
IOhannes
IRC
1:56:50
irssi that is
_hc
IRC
1:56:50
so you are proposing that each time Pd loads a lib, it sets the
*_PATH, then unsets it?
IOhannes
IRC
1:57:13
i fail to see the difference between DYLD_LIBRARY_PATH and
DYLD_FALLBACK_PATH
1:57:36
and yes, this is what my proposal is about
_hc
IRC
1:57:47
I don't quite understand it, but it seems to be related to the
"install name" thing
IOhannes
IRC
1:58:39
the big pro (imho), is, that it allows devs to work as they are used
to work with dylibs, and not invent some ramba zama magic dlopen()
hack for a standard problem
1:59:14
but don't my DYLD_LIBRARY_PATH scripts work for you?
_hc
IRC
2:00:05
some dlopen hack like externals?
IOhannes
IRC
2:00:54
no, an external is a plugin, and dlopen() is designed for plugins
2:01:52
in contrast, the automatic dynamic linker is designed for dynamic
sharing of code
2:02:07
which is what we want when we talk about "code-sharing"
2:02:16
and not: plugins
2:02:45
bot are closely related on a technical level, but this doesn't make
them interchangeable
2:03:27
s/bot/both/
2:03:36
is on a bad connection
2:05:12
according to http://msdn.microsoft.com/en-us/library/7d83bc18%28v=VS.100%29.aspx
, %PATH% is the envvar to use to trick the w32 linker
2:15:21
pob [~pob at gar31-4-82-240-199-62.fbx.proxad.net] entered the room.
_hc
IRC
3:07:09
IOhannes: what about when the libdir's .dylib is linked to other libs?
3:07:22
say /sw/bin/libspeex.dylib
3:09:59
r33p left the room (quit: Quit: ronflette mode).
IOhannes
IRC
3:14:56
what about it?
_hc
IRC
3:15:17
I feel like that's where there will be problems
IOhannes
IRC
3:15:26
why?
3:15:37
olsen [~sesselast at xdsl-188-155-179-205.adslplus.ch] entered the room.
IOhannes
IRC
3:15:51
maybe this is unclear: i'm talking about _adding_ to LD_LIBRARY_PATH,
not about replacing it
3:16:12
the dynamic linker will then additionally search in yet another
directory
_hc
IRC
3:16:14
did you run your tests on 10.4?
3:16:17
or just build them?
IOhannes
IRC
3:16:25
both
_hc
IRC
3:16:47
hmm, my decision on this was informed by some experience a long time
ago that i can't remember
3:17:00
so I feel like something is not going to work with this
IOhannes
IRC
3:18:08
maybe it was something trivial back then
_hc
IRC
3:18:48
with luck...
3:20:18
IOhannes: this project was already setup with this idea in mind, so
I'm trying it: github.com/pd-projects/tkwidgets
3:20:25
it has shared code in tkwidgets.c
IOhannes
IRC
3:22:16
i cannot checkout with my uplink right now...
_hc
IRC
3:22:41
wow, that's slow
IOhannes
IRC
3:24:26
what? my connection or your download speed?
_hc
IRC
3:25:06
your uplink
IOhannes
IRC
3:28:27
i know; officially i'm without internet at home
_hc
IRC
3:32:09
IOhannes: the test thing doesn't build for me on 10.5: http://pastebin.com/
3:32:24
http://pastebin.com/reTPU8MR
IOhannes
IRC
3:33:32
try "make distclean & autoreconf -fiv && ./configure && make"
3:33:42
(and get the ampersands rights)
_hc
IRC
3:34:57
oy, there were go: make distclean; /sw/bin/autoreconf-2.63 -fiv && ./
configure && make
3:36:18
IOhannes: so you link it normally, without DYLD_LIBRARY_PATH?
3:36:26
then set DLYD_ later?
3:36:50
lorenzosu [~lorenzosu at ppp-79-0.24-151.libero.it] entered the room.
IOhannes
IRC
3:36:58
DYLD_LIBRARY_PATH is an environment variable that allows you to tweak
the dynamic linker at runtime
_hc
IRC
3:37:19
I know
IOhannes
IRC
3:37:38
"dynamic linker at runtime" means: the piece of software that resolves
dynamic libraries when you run your executable
3:37:59
so DYLD_LIBRARY_PATH is never set in the compilation/linking stage
_hc
IRC
3:38:11
does it affect the linkign is my question
IOhannes
IRC
3:38:14
only when you run the actual program
3:38:21
which linking?
3:39:27
(there's the gcc linking the object files into a binary and adding
information on where to find external references)
3:39:58
(and theres the dynamic linker that resolves the external references
when you run the binary)
3:40:11
the first thing happens on the dev machine
3:40:19
the second thing happens on the deployment machine
3:40:31
DYLIB_LIBRARY_PATH has nothing to do with the first stop
3:40:44
it's _only_ about deployment
_hc
IRC
3:41:08
can you make libtool show you its command line?
3:41:22
I can't get this workign in the Makefile yet
IOhannes
IRC
3:41:28
usually it print's out the cmdline
_hc
IRC
3:42:25
this seems incomplete to me: libtool: link: gcc -g -O2 -o .libs/test
test.o libtest/.libs/libtest.dylib
3:42:42
oh wait, wrong one
3:43:02
that's the one: /bin/sh ./libtool --tag=CC --mode=link gcc -g -O2 -
module -avoid-version -shared -shrext .pd_darwin libtest/libtest.la -
o test.la -rpath /usr/local/lib test_la-test.lo
IOhannes
IRC
3:43:40
the line following that, is the actual gcc call
3:43:54
> gcc -Wl,-undefined -Wl,dynamic_lookup -o .libs/test.pd_darwin -
bundle .libs/test_la-test.o libtest/.libs/libtest.0.0.0.dylib
_hc
IRC
3:43:55
libtool: link: gcc -dynamiclib -Wl,-undefined -Wl,dynamic_lookup -
o .libs/libtest.0.dylib .libs/libtest.o -O2 -install_name /usr/
local/lib/libtest.0.dylib -compatibility_version 1 -current_version
1.0 -Wl,-single_module
3:43:56
yup
3:44:15
it has to have an install_name...
IOhannes
IRC
3:44:20
so it sets the install_name
_hc
IRC
3:44:22
and version
3:44:23
yup
IOhannes
IRC
3:44:39
but you can override that with DYLIB_LIBRARY_PATH
_hc
IRC
3:44:53
and a .dylib won't work without a install_name
3:44:55
that's the pisser
IOhannes
IRC
3:44:59
the libtool libraries are supposed to be as standard as can be
_hc
IRC
3:45:00
even tho you can override it
IOhannes
IRC
3:45:10
so what's the problem then?
_hc
IRC
3:45:19
that's probably what got me in the past
IOhannes
IRC
3:45:55
because our "overriding" is really only telling the linker to search
for all libraries in an additional place
3:46:28
it's not forcing anything (apart from a certain search _order_)
3:46:40
i have no clue what apple want the install_name to do
_hc
IRC
3:49:37
got it!
3:49:43
well, it built
3:50:11
doh!
3:50:23
the install_name is what is used in the linking
3:51:09
http://pastebin.com/Anf0BnUf
IOhannes
IRC
3:51:10
in which linking?
_hc
IRC
3:51:24
where would you do the DYLD_LIBRARY_PATH linking?
IOhannes
IRC
3:52:07
in the pd-loader
3:52:44
oiata [~oiata at cable-86-56-113-6.cust.telecolumbus.net] entered the room.
_hc
IRC
3:52:52
right, where in that process?
IOhannes
IRC
3:54:55
s_loader.c:172 (push (DY)LD_LIBRARY_PATH)
3:55:13
s_loader.c:173 pop (DY)LD_LIBRARY_PATH
3:55:25
s_loader.c:183: push PATH
3:55:33
s_loader.c:184: pop PATH
3:55:54
oops, s_loader.c:185: pop PATH
3:57:35
and actually it should be done for the scheduler-loader as well
(s_loader.c:246)
3:57:42
and for pd~
3:57:58
i wish miller had created a dlopen() wrapper
_hc
IRC
3:59:23
just read your email on this, I didn't think about including random
other libs, that's true it should work also
3:59:30
that would be very helpful on Windows and Mac OS X
_hc
IRC
4:06:23
IOhannes: strange, it seems to work without the DYLD_LIBRARY_PATH set
IOhannes
IRC
4:07:10
what are you doing?
4:07:40
maybe you have the library sitting around somewhere
_hc
IRC
4:07:56
to link the dylib:
4:07:57
-dynamiclib -undefined dynamic_lookup -install_name $(SHARED_LIB) -
compatibility_version 1 -current_version 1.0
4:08:05
to link the objects:
4:08:15
./$(SHARED_LI
4:08:27
cc -arch ppc -arch i386 -arch x86_64 -mmacosx-version-min=10.4 -
bundle -bundle_loader /Applications/Pd-extended.app/Contents/Resources/
bin/pd -undefined dynamic_lookup -o "text.pd_darwin" "text.o" -lc -L/
sw/lib ./libtkwidgets.dylib
4:11:19
here's the actual dylib link:
4:11:19
cc -arch ppc -arch i386 -arch x86_64 -mmacosx-version-min=10.4 -
dynamiclib -undefined dynamic_lookup -install_name libtkwidgets.dylib -
compatibility_version 1 -current_version 1.0 -o libtkwidgets.dylib
tkwidgets.o -lc -L/sw/lib
4:12:29
IOhannes: I figured it out, I was starting pd in the tkwidgets/
folder, that's how it found it
IOhannes
IRC
4:15:39
the only problem i have right now, is that it is crashing on linux
_hc
IRC
4:16:34
IOhannes: does the lib need a versioned name in Linux? ie. can we just
have libtkwidgets.so or must it be libtkwidgets.so.1.
IOhannes
IRC
4:18:52
i _think_ the latter (though i'm not 100% sure)
4:19:27
ah, i think if the library has no so_version, you might get away
with .sl
4:19:31
.so, i meant
More information about the Pd-dev
mailing list