[PD-dev] how to load shared code for libraries
Hans-Christoph Steiner
hans at at.or.at
Mon Sep 26 18:52:08 CEST 2011
IOhannes and I were discussing how to load shared code for a libdir on
IRC. The sticking point is that Mac OS X seems to require dylibs to
have a hard-coded path. IOhannes posted a test lib to try to work out
how to do it using DYLD_LIBRARY_PATH:
http://iem.at/~zmoelnig/OSX/test.tgz
So here are my results from Mac OS X 10.5.8:
creating a [test] object:
-------------------------
/private/tmp/test/tmp/test.pd_darwin: dlopen(/private/tmp/test/tmp/
test.pd_darwin, 10): no suitable image found. Did find:
/private/tmp/test/tmp/test.pd_darwin: unknown required load
command 0x80000022
test
... couldn't create
hans at dhcp-10 tmp $ ~/code/pd-extended.git/src/pd -noprefs -lib test
-------------------------------------------------------------------
./test.pd_darwin: dlopen(./test.pd_darwin, 10): no suitable image
found. Did find:
./test.pd_darwin: unknown required load command 0x80000022
test: can't load library
Here is the IRC transcript, for those who might be interested:
--------------------------------------------------------------
but it seems, you are trying to find arguments against single-object-
per-lib
11:23:51
buscon [~chatzilla at orange.demos-europe.eu] entered the room.
_hc
IRC
11:24:47
the only thing they currently fix IMHO is the hexloader issue and
shared code
11:25:05
both of these can be handled in the libdir style, but its just not
implemented yet...
11:25:26
I have an idea that should work for the shared code
IOhannes
IRC
11:26:28
?
_hc
IRC
11:27:17
just have an easy way for each class to call dlopen on a shared
library as part of setup()
11:27:45
based on c_externdir
IOhannes
IRC
11:28:54
uäh
11:29:12
dlopen() is not really about sharing code, but about externalizing it
_hc
IRC
11:29:22
eh?
IOhannes
IRC
11:29:24
(plugins)
_hc
IRC
11:29:33
its about opening a dynamic library
11:29:38
hence d-l-open
IOhannes
IRC
11:29:56
exactly, and this is not what you usually do when you want to share code
11:30:04
mamalala left the room (quit: Ping timeout: 256 seconds).
_hc
IRC
11:30:13
you mean you want it private to the project
11:30:27
when I share code, it is very much what I usually od
IOhannes
IRC
11:30:58
nah, i mean when i hear s"sharing code" i think "dynamically linking
against a library"
11:31:08
this does not require to dlopen() a file
_hc
IRC
11:31:22
well, it does on Mac OS X for deployment
11:31:35
because dylibs have static paths coded in them
IOhannes
IRC
11:31:36
it requires to tell the linker that i want to use a library
_hc
IRC
11:31:40
that doesn't work for externals
11:31:45
you have to load them directly with dlopen
11:32:09
mamalala [~mamalala at 89.204.137.162] entered the room.
IOhannes
IRC
11:32:14
Gem uses a big shared library (openGL) and i haven't seen it
dlopen()end on any platform (yet)
_hc
IRC
11:32:17
so either use a hard-coded path or dlopn
IOhannes
IRC
11:32:31
argh
_hc
IRC
11:32:33
yes, because Gem has a hardcoded path to OpenGL
11:32:43
and it is in the same path on all Mac O SX systems
IOhannes
IRC
11:33:04
so it won't find libGL if it was in /usr/local/lib(?
_hc
IRC
11:33:35
that's the dynamic lookup of symbols in the lib path
IOhannes
IRC
11:33:44
so?
_hc
IRC
11:34:06
how are you going to put ~/Library/Pd/Gem into the lib path?
IOhannes
IRC
11:34:24
i think it would be nice if the loader would:
11:34:42
find Gem.pd_darwin in ~/Library/Pd/Gem
11:34:53
then add ~/Library/Pd/Gem to the LD_LIBRARY_PATH
11:35:05
then delopen() Gem.pd_darwin
11:35:19
emoc left the room (quit: Quit: .:*%##~¤#*° °° ° °).
IOhannes
IRC
11:35:20
wants to say dlopen(), of course
11:35:27
and finally restore the LD_LIBRARY_PATH
_hc
IRC
11:35:31
LD_LIBRARY_PATH is global
11:35:41
that sounds like a recipe for disaster
IOhannes
IRC
11:35:45
i don't see a problem
11:35:49
tonussi left the room (quit: Quit: Ex-Chat).
_hc
IRC
11:36:03
same issues as Pd-extended has loading all libs by default
IOhannes
IRC
11:36:06
you restore LD_LIBRARY_PATH to the original value after trying to
dlopen() Gem.pd_darwin
_hc
IRC
11:36:07
namespace conflicts
11:36:15
why not just dlopen() then?
11:36:30
buscon left the room (quit: Quit: ChatZilla 0.9.87 [Firefox
6.0.2/20110902133214]).
IOhannes
IRC
11:36:35
because the logic needs to be implemented only once
_hc
IRC
11:36:40
if you know in advance whhich DLL you need, why bother searching fo rit
11:36:46
same with the dlopen() logic
IOhannes
IRC
11:36:56
and you don't have to work with error-prone function pointers
11:37:19
how is your development workflow (on systems that do not have weak-
linking built-in)?
_hc
IRC
11:37:24
hmm, I think function pointers work quite well
11:37:39
cd mylibdir && make
IOhannes
IRC
11:37:40
it works fine, if you don't have to repeat the logic
_hc
IRC
11:37:41
and it'll work
IOhannes
IRC
11:37:48
_that_ sounds like disaster
_hc
IRC
11:37:57
what?
IOhannes
IRC
11:38:24
having _all_ devs have to implement their function pointers again and
again
11:38:43
"cd mylibdir && make" is approximately how you build each and every
external
11:38:50
i don't see how this is related to dlopen()
_hc
IRC
11:39:06
you asked about the dev workflow
IOhannes
IRC
11:39:07
and then dlopen() is different on w32 and un*x
11:39:16
_hc
IRC
11:39:18
MinGW provides libdl
IOhannes
IRC
11:39:26
vendor lock in
_hc
IRC
11:39:37
lol
IOhannes
IRC
11:39:51
back to "development workflow"
_hc
IRC
11:39:52
except you have the sources and right to do whatever you want with it
11:40:05
yes, you are correct, I don't care to support MSVC
IOhannes
IRC
11:40:07
like with "libreoffice"?
11:40:21
back to "development workflow"
_hc
IRC
11:40:24
its too much work and not worth it
11:40:32
but I'd be happy if someone else wants to do it
11:40:36
its all free software
IOhannes
IRC
11:40:44
to do what?
_hc
IRC
11:40:55
handle MSVC compatibility
IOhannes
IRC
11:41:14
after deliberately breaking it by depending on dlopen()?
11:41:26
i still don't see the problem with LD_LIBRARY_PATH
_hc
IRC
11:41:30
someone could build libdl using MSVC, I am sure
IOhannes
IRC
11:41:48
but why would anybody want to do that, if they have LoadLibrary
11:41:59
which is not really better or worse
_hc
IRC
11:42:16
libdl for Windwos is a wrapper on LoadLibrary
11:42:23
Mac OS X did the same thing, albeit officially
IOhannes
IRC
11:42:27
so?
_hc
IRC
11:42:39
it means you don't need #ifdef _WIN32
11:42:41
very nice
IOhannes
IRC
11:43:06
i don't want to "#ifdef _WIN32" because i don't see why i would need
any platform dependent code in my setup()
_hc
IRC
11:43:12
IOhannes: you want to add the currently loading lib to
LD_LIBRARY_PATH, and then remove it?
IOhannes
IRC
11:43:13
_very_ nice
IOhannes
IRC
11:43:29
_hc: that's about it
_hc
IRC
11:43:40
won't work on Mac OS X because of the hardcoded paths issue
11:43:44
there are two paths:
11:43:53
library id, i.e... "myself"
11:43:59
and libraries I am linked ot
IOhannes
IRC
11:44:47
but iirc, those hardcoded paths can be relative, so on OSX we don't
even need the LD_LIBRARY_PATH
_hc
IRC
11:45:08
only relative to the Pd exe
11:45:34
then libs would only work if installed into ~/Library/Pd and Pd
installed into /Applications with the same exact name
11:45:38
all dev builds would break
11:46:01
as would the ability to run Pd from any folder in Mac OS X, which is
normal
11:46:07
loadbang left the room (quit: Read error: Operation timed out).
IOhannes
IRC
11:46:28
but what does "man dyld" mean with: "DYLD_LIBRARY_PATH This is a colon
separated list of directories that contain libraries. The dynamic
linker searches these directories before it searches the default
locations for libraries. It allows you to test new versions of
existing libraries."
_hc
IRC
11:46:31
its not relative paths really, there is a single variable
@executable_path, which is the path to pd
11:46:40
loadbang [~loadbang at host86-177-236-179.range86-177.btcentralplus.com]
entered the room.
_hc
IRC
11:46:51
sure, that is also the case
11:46:58
that doesn't contradict what I say
IOhannes
IRC
11:47:31
for me it sounds like this is the mechanism i propose with
manipulating LD_LIBRARY_PATH
_hc
IRC
11:48:03
right, but libs in DYLD_LINRARY_PATH still need to have their ID as a
hardcoded path
11:48:16
so export = DYLD_LIBRARY_PATH=/foo
11:48:27
the library is linked to be /foo/mylib.dylib
IOhannes
IRC
11:49:00
i don't exactly get what "is linked to be" means here
11:49:50
are you trying to say, that there is a conflict between the libraries
"real" and "self" path, and that the "self" path (the one the library
believes it resides in), has to be correct?
_hc
IRC
11:49:59
they are called "install names"
IOhannes
IRC
11:50:03
wonders whether he makes sense
_hc
IRC
11:50:54
yes, the "install name" must be the actual path it resides in
11:50:57
or it doesn't work
11:51:43
you can see it by doing "otool -L /usr/lib/libc.dylib"
11:52:12
_ /usr/lib/libc.dylib:
_ /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 111.1.7)
_ /usr/lib/system/libmathCommon.A.dylib (compatibility version
1.0.0, current version 292.4.0)
11:52:20
the first line is the install name
11:52:22
the others the links
IOhannes
IRC
11:52:57
but if i copy libc.dylib to $(PWD) and do a "otool -L libc.dylib", the
"install name" says:
11:53:00
libc.dylib:
11:53:53
(forget the trailing ":")
_hc
IRC
11:56:02
oops, bad example
11:56:26
otool -L /usr/lib/libfcgi.dylib
11:56:32
tonussi [~lucaspt at 177.40.10.96] entered the room.
_hc
IRC
11:56:41
_/usr/lib/libfcgi.dylib:
_ /usr/lib/libfcgi.0.dylib (compatibility version 1.0.0, current
version 1.0.0)
_ /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 111.0.0)
_ /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current
version 1.0.0)
11:56:46
libc I guess is special
11:56:52
libc is weird on Mac OS X
11:57:14
hans at dhcp-10 pd-extended.git $ ls -l /usr/lib/libc.dylib
lrwxr-xr-x 1 root wheel 15 2011-05-02 04:59 /usr/lib/libc.dylib ->
libSystem.dylib
11:57:43
actually, libc is actually libSystem, and you can see the first entry
is libSystem
11:57:47
that's the ID
IOhannes
IRC
11:58:07
i'll have to try this out
_hc
IRC
11:58:12
please do
IOhannes
IRC
11:58:15
is sorry for being a disbeliever
_hc
IRC
11:58:21
it sucks, but its true
IOhannes
IRC
12:09:15
hmm, i cannot reproduce this
12:10:11
i created a library libtest, and compiled it 2 times with different
flags, so it's sole function will print something different depending
on which compilation you are using
12:10:26
then i created a binary, that uses this library
12:10:52
with DYLD_LIBRARY_PATH, i can chose which version of the library is used
12:11:18
the .dylibs are anywhere but in the path where they are expected
(which would be /usr/local/lib)
IOhannes
IRC
12:17:36
now trying with an external
_hc
IRC
12:40:09
IOhannes: any luck?
12:40:23
dto left the room (quit: Quit: Leaving.).
IOhannes
IRC
12:40:58
http://iem.at/~zmoelnig/OSX/test.tgz
12:41:25
i can tell an external which dylib to use by setting DYLD_LIBRARY_PATH
12:41:37
i have to run now
----------------------------------------------------------------------------
"[T]he greatest purveyor of violence in the world today [is] my own
government." - Martin Luther King, Jr.
More information about the Pd-dev
mailing list