Linking to Python framework on OSX

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Post Reply
Message
Author
User avatar
Vezzra
Release Manager, Design
Posts: 5450
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Linking to Python framework on OSX

#1 Post by Vezzra »

In an attempt to fix bug report #553 on SF I've added the following to HumanClientApp::StartServer, before the server process gets created:

Code: Select all

#ifdef FREEORION_MACOSX
    // On OSX set environment variable DYLD_LIBRARY_PATH to python framework folder
    // bundled with app, so the dynamic linker uses the bundled python library.
    // Otherwise the dynamic linker will look for a correct python lib in system
    // paths, and if it can't find it, throw an error and terminate!
    // Setting environment variable here, spawned child processes will inherit it.
    setenv("DYLD_LIBRARY_PATH", GetPythonHome().string().c_str(), 1);
#endif
Actually, I just copied that from ServerApp::CreateAIClients. This piece of code makes sure that the Python framework shipped with the FO app bundle is used instead of one installed on the system, as this can cause all kind of problems, probably also those described in the bug report mentioned above.

Now, although that approach works, I don't like it - at all. It's just, ugh, ugly IMO. The problem stems from shared library id name of the Python framework, as output by otool:

Code: Select all

$ otool -D PythonPython (architecture ppc):
/Library/Frameworks/Python.framework/Versions/2.7/Python
Python (architecture i386):
/Library/Frameworks/Python.framework/Versions/2.7/Python
Obviously, this library id name is incorporated into the server/ai client executables by the linker, again as output by otool:

Code: Select all

$ otool -L freeoriond
freeoriond:
	/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
	@executable_path/../SharedSupport/freeorioncommon.dylib (compatibility version 1.0.0, current version 1.0.0)
	@executable_path/../SharedSupport/freeorionparse.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)

Code: Select all

$ otool -L freeorionca
freeorionca:
	/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
	@executable_path/../SharedSupport/freeorioncommon.dylib (compatibility version 1.0.0, current version 1.0.0)
	@executable_path/../SharedSupport/freeorionparse.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)
Which leads to the dynamic linker looking for the Python framework in /Library/Frameworks instead of @executable_path/../Frameworks when these executables are launched. With freeorioncommon.dylib and freeorionparse.dylib that works, because they are build by the FO Xcode project and there I can set the so called "Dynamic Library Install Name" accordingly. But how can I make it so that this also works that way with the Python framework bundled with the FO app? So that the shared library id name incorporated into the server and ai client looks like this:

Code: Select all

@executable_path/../Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
I'm pretty sure if I get that done, I could throw out the setenv statement prior to the server/ai client process creation.

Anyone any idea? Adrian perhaps?

User avatar
adrian_broher
Programmer
Posts: 1155
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Linking to Python framework on OSX

#2 Post by adrian_broher »

Vezzra, have a look at this
http://www.dribin.org/dave/blog/archive ... /15/rpath/
and this
https://wincent.com/wiki/@executable_pa ... [email protected]

It's a bit of fiddling and you may need to delete the Freeorion.bundle manually when testing.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 5450
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Linking to Python framework on OSX

#3 Post by Vezzra »

Thanks for the links!

I've already taken a look, question: All these articles (apparently) assume that the dynamic library id name is already correctly set (using @executable_path or @rpath instead of an absolute path). Of course, in that case when I build my executable it gets this correct path. The problem is, the Python shared library contained in the Python framework has been built with a id name of "/Library/Frameworks/Python.framework/Versions/2.7/Python".

Now of course I can change the dynamic library id name of the Python library file, but that's not an optimal solution. Because in that case I've to remember every time I upgrade to a more current version of the Python framework to do that again. If I forget, the next FO app I build will again load Python libraries installed on the system instead of the bundled one, resulting in all kind of troubles, bug reports and the necessity to fix the mess.

I want to avoid that, so I'm looking for a solution that enables me to solve that automatically, some way to ensure that when building the AI clients and the server, not the absolute path of the Python shared library id name gets used, but @executable_path or @rpath. But I haven't found anything in the articles you linked that deals with that part of the problem... maybe I've overlooked something.

User avatar
adrian_broher
Programmer
Posts: 1155
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Linking to Python framework on OSX

#4 Post by adrian_broher »

Where do you get the python frameworks from? Did you build it by yourself? Or is it released somewhere?
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 5450
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Linking to Python framework on OSX

#5 Post by Vezzra »

adrian_broher wrote:Where do you get the python frameworks from? Did you build it by yourself? Or is it released somewhere?
http://python.org/download/

No, I don't build them myself. Doing that would be even more of an effort than trying to remember to change the dynamic library id name.

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Linking to Python framework on OSX

#6 Post by Dilvish »

Vezzra wrote:Now of course I can change the dynamic library id name of the Python library file, but that's not an optimal solution. Because in that case I've to remember every time I upgrade to a more current version of the Python framework to do that again. If I forget, the next FO app I build will again load Python libraries installed on the system instead of the bundled one, resulting in all kind of troubles, bug reports and the necessity to fix the mess.
a new 2.7.x framework coming out is relatively rare isn't it? When one does, I could imagine that if you forget about it rebuilding FO could create a ton of error spam, but would it really be that messy to fix at that point? I don't mean to tell you how to do it, since I really don't know the process, but, erm, I can't help but stick my nose into issues sometimes :oops: :lol:
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Vezzra
Release Manager, Design
Posts: 5450
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Linking to Python framework on OSX

#7 Post by Vezzra »

Dilvish wrote:a new 2.7.x framework coming out is relatively rare isn't it? When one does, I could imagine that if you forget about it rebuilding FO could create a ton of error spam, but would it really be that messy to fix at that point?
You're right of course, it's just that I really dislike patching things up manually that way. And don't forget, considering it'll probably take us another decade to reach 1.0, we'll probably have to upgrade to Python 3 (or even 4? :shock:) at some point... :mrgreen:
I don't mean to tell you how to do it, since I really don't know the process, but, erm, I can't help but stick my nose into issues sometimes :oops: :lol:
That's what friends are for, aren't they? ;)

Post Reply