Arch compile error in Python[ServerFramework|AI].cpp

Questions, problems and discussion about compiling FreeOrion.

Moderator: Oberlus

Message
Author
Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Arch compile error in Python[ServerFramework|AI].cpp

#1 Post by Chriss »

Code: Select all

[chris@desk build]$ export LANG=C; make -j1 -k
Building v0.4.4+  build 2015-05-24.b39cbba
[  0%] Built target freeorionversion
[ 25%] Built target freeorioncommon
[ 40%] Built target GiGi
[ 40%] Built target GiGiSDL
[ 62%] Built target freeorionparse
[ 63%] Building CXX object server/CMakeFiles/freeoriond.dir/__/python/server/PythonServerFramework.cpp.o
/home/chris/freeorion-workspace/freeorion.git/python/server/PythonServerFramework.cpp: In function 'bool PythonInit()':
/home/chris/freeorion-workspace/freeorion.git/python/server/PythonServerFramework.cpp:98:23: error: 'initfo_logger' was not declared in this scope
         initfo_logger();              // allows the "fo_logger" C++ module to be imported within Python code
                       ^
/home/chris/freeorion-workspace/freeorion.git/python/server/PythonServerFramework.cpp:107:23: error: 'initfreeorion' was not declared in this scope
         initfreeorion();
                       ^
server/CMakeFiles/freeoriond.dir/build.make:284: recipe for target 'server/CMakeFiles/freeoriond.dir/__/python/server/PythonServerFramework.cpp.o' failed
make[2]: *** [server/CMakeFiles/freeoriond.dir/__/python/server/PythonServerFramework.cpp.o] Error 1
make[2]: Target 'server/CMakeFiles/freeoriond.dir/build' not remade because of errors.
CMakeFiles/Makefile2:309: recipe for target 'server/CMakeFiles/freeoriond.dir/all' failed
make[1]: *** [server/CMakeFiles/freeoriond.dir/all] Error 2
[ 63%] Building CXX object client/AI/CMakeFiles/freeorionca.dir/__/__/AI/PythonAI.cpp.o
/home/chris/freeorion-workspace/freeorion.git/AI/PythonAI.cpp: In constructor 'PythonAI::PythonAI()':
/home/chris/freeorion-workspace/freeorion.git/AI/PythonAI.cpp:261:29: error: 'initfreeOrionLogger' was not declared in this scope
         initfreeOrionLogger();          // allows the "freeOrionLogger" C++ module to be imported within Python code
                             ^
/home/chris/freeorion-workspace/freeorion.git/AI/PythonAI.cpp:262:34: error: 'initfreeOrionAIInterface' was not declared in this scope
         initfreeOrionAIInterface();     // allows the "freeOrionAIInterface" C++ module to be imported within Python code
                                  ^
client/AI/CMakeFiles/freeorionca.dir/build.make:100: recipe for target 'client/AI/CMakeFiles/freeorionca.dir/__/__/AI/PythonAI.cpp.o' failed
make[2]: *** [client/AI/CMakeFiles/freeorionca.dir/__/__/AI/PythonAI.cpp.o] Error 1
make[2]: Target 'client/AI/CMakeFiles/freeorionca.dir/build' not remade because of errors.
CMakeFiles/Makefile2:360: recipe for target 'client/AI/CMakeFiles/freeorionca.dir/all' failed
make[1]: *** [client/AI/CMakeFiles/freeorionca.dir/all] Error 2
Linking CXX executable ../../freeorion
CMakeFiles/freeorion.dir/chmain.cpp.o: In function `mainConfigOptionsSetup(std::vector<std::string, std::allocator<std::string> > const&)':
chmain.cpp:(.text+0xe1d): undefined reference to `FreeOrionVersionString()'
chmain.cpp:(.text+0x1a29): undefined reference to `FreeOrionVersionString()'
CMakeFiles/freeorion.dir/chmain.cpp.o: In function `mainSetupAndRun()':
chmain.cpp:(.text+0x37fb): undefined reference to `FreeOrionVersionString()'
CMakeFiles/freeorion.dir/__/__/UI/IntroScreen.cpp.o: In function `IntroScreen::IntroScreen()':
IntroScreen.cpp:(.text+0x354d): undefined reference to `FreeOrionVersionString()'
collect2: error: ld returned 1 exit status
client/human/CMakeFiles/freeorion.dir/build.make:1720: recipe for target 'freeorion' failed
make[2]: *** [freeorion] Error 1
make[2]: Target 'client/human/CMakeFiles/freeorion.dir/build' not remade because of errors.
CMakeFiles/Makefile2:413: recipe for target 'client/human/CMakeFiles/freeorion.dir/all' failed
make[1]: *** [client/human/CMakeFiles/freeorion.dir/all] Error 2
make[1]: Target 'all' not remade because of errors.
Makefile:136: recipe for target 'all' failed
make: *** [all] Error 2
make: Target 'default_target' not remade because of errors.
Any Ideas?
Attached patches are released under GPL 2.0 or later.

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Arch compile error in Python[ServerFramework|AI].cpp

#2 Post by Dilvish »

In one of your other recent posts about cmake and Version.cpp you mentioned something about symlinking 'python2' to python3-- that may have worked for that one particular bit of scripting, but in general python2.7 scripts will simply not be compatible with python3. Also, the particular things being complained of in your logs, calls to initfo_logger() and initfreeOrionLogger() are provided via Boost macros, which apparently are in some ways specific to python 2.x, or at least differ between python2.x and python3.x Here is one thread from a month ago asserting (at least for their build) that boost 1.57 had hidden dependencies on python2.7. Here is an actual boost python dev thread through that indicates it is possible (and how) to build separate boost versions for python 2.7 and python3. I suggest you look more into that & would appreciate it if you could let us know what you come up with. Once you have a python2.7 specific boost to build with, here is an alias I use for specifying building fo with a particular boost version:

Code: Select all

alias CMBR='cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INCLUDE_PATH=/path/to/boost_1_57_0/install/include -DCMAKE_LIBRARY_PATH=/path/to/boost_1_57_0/install/lib '
I see there is a PEP0394 regarding how to manage python2 and python3 coexistence which mentions it was partly prompted by arch linux. Since you are running into this at compile/link time it seems we may need to make some changes as well, but from reading the pep it sounds highly unlikely that simply symlinking 'python2' to python3 is going to be part of the solution. Reading some threads, like this one, it sounds like for some people it worked to uninstall all python versions, then install 3.3, then install 2.7.

For reference, here is the comparable PEP397 for Windows.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Arch compile error in Python[ServerFramework|AI].cpp

#3 Post by Chriss »

To clarify some things: Arch, or my install at least, has python2 (2.7.9) and python3 (3.4.3) both.

Code: Select all

[chris@desk ~]$ ls -l /bin/python*
lrwxrwxrwx 1 root root     7 25. Mär 18:30 /bin/python -> python3
lrwxrwxrwx 1 root root     9 11. Dez 05:51 /bin/python2 -> python2.7
-rwxr-xr-x 1 root root  6224 11. Dez 05:51 /bin/python2.7
-rwxr-xr-x 1 root root  1681 11. Dez 05:51 /bin/python2.7-config
lrwxrwxrwx 1 root root    16 11. Dez 05:51 /bin/python2-config -> python2.7-config
-rwxr-xr-x 1 root root    61 28. Mär 18:05 /bin/python2-pyuic5
lrwxrwxrwx 1 root root     9 25. Mär 18:30 /bin/python3 -> python3.4
-rwxr-xr-x 2 root root 10440 25. Mär 18:30 /bin/python3.4
lrwxrwxrwx 1 root root    17 25. Mär 18:30 /bin/python3.4-config -> python3.4m-config
-rwxr-xr-x 2 root root 10440 25. Mär 18:30 /bin/python3.4m
-rwxr-xr-x 1 root root  3107 25. Mär 18:30 /bin/python3.4m-config
lrwxrwxrwx 1 root root    16 25. Mär 18:30 /bin/python3-config -> python3.4-config
lrwxrwxrwx 1 root root    14 25. Mär 18:30 /bin/python-config -> python3-config
What I meant is that python3 is the default if you execute "python". The python script in the other thread specifies (first line #/bin/python) no specific python version, which is why it is executed by python3 on my system. If I change the first line of the script to #/bin/python2 it is executed via python2 and everything works. I did not change or create any symlinks.

I know enough about python to know that 2 and 3 are not compatible. I could've fixed the print statements myself, probably, I am simply not sure about the general picture. And more importantly, whether that fix would by python2 compatible.

About the issue here - did that stuff change recently? Is this because of my newer boost version, because of the python version boost is built with? Did that maybe change for my system? Some changes there? I've had both python2 and python3 on my system for quite some time, this is the first time that an issue like this pops up. I'll investigate a bit more, but any hints are appreciated.
Attached patches are released under GPL 2.0 or later.

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Arch compile error in Python[ServerFramework|AI].cpp

#4 Post by Chriss »

Dilvish wrote:I see there is a PEP0394 regarding how to manage python2 and python3 coexistence which mentions it was partly prompted by arch linux. Since you are running into this at compile/link time it seems we may need to make some changes as well, but from reading the pep it sounds highly unlikely that simply symlinking 'python2' to python3 is going to be part of the solution. Reading some threads, like this one, it sounds like for some people it worked to uninstall all python versions, then install 3.3, then install 2.7.

For reference, here is the comparable PEP397 for Windows.
Basically, that PEP only suggests a way of handling the "which version gets called" situation from the other thread. A script which is not python3 source compatible (like the one in the other thread) should explicitly call python2 in the shebang line, which is the change I've made to make it work. So this PEP should not have anything to do with the issue here, at least not at first glance.
Attached patches are released under GPL 2.0 or later.

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Arch compile error in Python[ServerFramework|AI].cpp

#5 Post by Chriss »

Code: Select all

[chris@desk build]$ rm -r * && cmake ../freeorion.git/ && make -j8 -k
-- Setting build type to 'Release' as none was specified.
-- The C compiler identification is GNU 5.1.0
-- The CXX compiler identification is GNU 5.1.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found PythonInterp: /usr/bin/python (found version "3.4.3") 
-- Boost version: 1.57.0
-- Found the following Boost libraries:
--   chrono
--   date_time
--   filesystem
--   signals
--   system
--   thread
--   log
-- Found JPEG: /usr/lib64/libjpeg.so  
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.8") 
-- Found PNG: /usr/lib64/libpng.so (found version "1.6.16") 
-- Found TIFF: /usr/lib64/libtiff.so (found version "4.0.3") 
-- Found OpenGL: /usr/lib64/libGL.so  
-- Found Freetype: /usr/lib64/libfreetype.so (found version "2.5.5") 
-- Configuring GiGi
-- Configuring GiGiSDL
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.28") 
-- checking for one of the modules 'sdl2'
-- Boost version: 1.57.0
-- Found the following Boost libraries:
--   chrono
--   date_time
--   filesystem
--   regex
--   serialization--   signals
--   system
--   thread
--   log
-- Configuring freeoriond
-- Boost version: 1.57.0
-- Found the following Boost libraries:
--   python
--   log
-- Found PythonLibs: /usr/lib64/libpython3.4m.so (found suitable version "3.4.3", minimum required is "2") 
-- Configuring freeorionca
-- Boost version: 1.57.0
-- Found the following Boost libraries:
--   python
--   log
-- Configuring freeorion
-- Found GLEW: /usr/include  
-- Found OpenAL: /usr/lib64/libopenal.so  
-- Found Ogg: /usr/lib64/libogg.so  
-- Found Vorbis: /usr/lib64/libvorbis.so  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/chris/freeorion-workspace/build
Scanning dependencies of target freeorionversion
Scanning dependencies of target GiGi
Scanning dependencies of target freeorionparse
Building v0.4.4+  build 2015-05-24.b39cbba
[  0%] Built target freeorionversion
Scanning dependencies of target freeorioncommon
[...]
So yeah... looks like arch's boost is compiled against python3... Could that be the issue?
Attached patches are released under GPL 2.0 or later.

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Arch compile error in Python[ServerFramework|AI].cpp

#6 Post by Vezzra »

Chriss wrote:About the issue here - did that stuff change recently?
That depends on what you mean by "recently". There has been some work done on the server-side Python interface at the beginning of March. However, the error also happens in the AI client Python interface, so I don't think it has anything to do with these changes.
Is this because of my newer boost version, because of the python version boost is built with? Did that maybe change for my system?
That would be my guess. Otherwise I'm pretty clueless, sorry...

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Arch compile error in Python[ServerFramework|AI].cpp

#7 Post by Vezzra »

Chriss wrote:So yeah... looks like arch's boost is compiled against python3... Could that be the issue?
I very much suspect so.

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Arch compile error in Python[ServerFramework|AI].cpp

#8 Post by Chriss »

Are you guys sure this is python related? I mean that's a C++ function call and neither I nor grep see where that function is defined...

Code: Select all

[chris@desk freeorion.git]$ grep -r initfo_logger .
./python/server/PythonServerFramework.cpp:        initfo_logger();              // allows the "fo_logger" C++ module to be imported within Python code
[chris@desk freeorion.git]$ grep -r initfreeorion .
./python/server/PythonServerFramework.cpp:        initfreeorion();
[chris@desk freeorion.git]$ grep -r initfreeOrionLogger .
./AI/PythonAI.cpp:        initfreeOrionLogger();          // allows the "freeOrionLogger" C++ module to be imported within Python code
[chris@desk freeorion.git]$ grep -r initfreeOrionAIInterface .
./AI/PythonAI.cpp:        initfreeOrionAIInterface();     // allows the "freeOrionAIInterface" C++ module to be imported within Python code
Shouldn't there be another reference somewhere in the git source tree?
Attached patches are released under GPL 2.0 or later.

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Arch compile error in Python[ServerFramework|AI].cpp

#9 Post by Dilvish »

Chriss wrote:Are you guys sure this is python related? I mean that's a C++ function call and neither I nor grep see where that function is defined......
Shouldn't there be another reference somewhere in the git source tree?
That is provided via a Boost python macro at line 36

Code: Select all

BOOST_PYTHON_MODULE(fo_logger) {
that is one of the reasons I was thinking you needed a different boost build.

From the Python C PI Reference, t also looks to me like we need to make some changes to how we invoke python from with FO. Right now when we load the python interpreter, only for MacOSX do we invoke Py_SetProgramName (setting it to the embedded python copy). It looks to me that for Windows and Linux we also need to start calling Py_SetProgramName to set the name to "python2", because the default is simply "python". Although the MacOSX code is also calling Py_SetPythonHome(), it looks to me like this shouldn't be necessary in general-- once Py_SetProgramName has been called the pythonhome info etc. should get derived from that. (Vezzra, have you tried that approach for MaxOSX?)

So, Chris, in addition to having a copy boost built for python2 (& telling cmake where to find it as I mentioned before), it looks to me like we'd need to add this Py_SetProgramName call to

Code: Select all

AI/PythonAI.cpp
python/server/PythonServerFramework.cpp
Chris, could you make a go at that and let us know if that does the trick?
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Arch compile error in Python[ServerFramework|AI].cpp

#10 Post by Chriss »

Hm... I'd really like to not have an extra boost build... Also since the Arch Package would need that too. It seems odd because older boost packages have the same issues. I don't think that python3 stuff is all that new in arch, though I do not know for sure. I'll do some more tests.

If that macro fails, shouldn't there be any error messages somewhere? How would I debug that?
Attached patches are released under GPL 2.0 or later.

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Arch compile error in Python[ServerFramework|AI].cpp

#11 Post by Vezzra »

Dilvish wrote:It looks to me that for Windows and Linux we also need to start calling Py_SetProgramName to set the name to "python2", because the default is simply "python".
Well, my PR#94 is a patch that introduces exactly that for Windows. I made the necessary changes so the code that until now only has been used on OSX now also is used on Windows.

On Windows however this thing with the Python executable works differently, from what I've seen when implementing that patch. There are no "python2" or "python2" aliases you can use, and FO on Windows does not use a Python executable at all, it uses a DLL ("Python27.dll") which is bundled with FO. Peeking at the logs revealed that therefore the Python program name isn't that of a Python interpreter executable, but that of the executable that loaded the DLL (FreeOrionD.exe and FreeOrionCA.exe in our case).
Although the MacOSX code is also calling Py_SetPythonHome(), it looks to me like this shouldn't be necessary in general-- once Py_SetProgramName has been called the pythonhome info etc. should get derived from that. (Vezzra, have you tried that approach for MaxOSX?)
In theory, you're maybe right, in practice I didn't ever bother to try, but just took the save road and called both (Py_SetProgramName and Py_SetPythonHome). As far as I remember, it took me several attempts to finally get FO to use the bundled Python instead of the system Python, every time it seemed to work, but didn't, it just seemed to work because I somehow managed to fool myself. It's tricky when the version of Python installed on your system is the same as the bundled one - it works, you look at the logs, see the expected version, are happy, unless someone else, who has a version of Python installed that causes issues, reports troubles, or you change something on your system that causes FO Python troubles. You go back, find out you fooled yourself, try to fix it, it works, but somehow you managed to fool yourself again (don't ask me how I did it :?), lather, rinse repeat until you're on the brink of insanity.

At some point I've been fed up, threw in all I could think of that might work, and that more or less solved the issues - on OSX that is. Of course I could try if taking out the call to Py_SetPythonHome would work, but, well, *gulp* ... *shudder* ...

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Arch compile error in Python[ServerFramework|AI].cpp

#12 Post by Vezzra »

Chriss wrote:Hm... I'd really like to not have an extra boost build... Also since the Arch Package would need that too. It seems odd because older boost packages have the same issues. I don't think that python3 stuff is all that new in arch, though I do not know for sure.
That may all be, but fact is that boost python has to be built/linked against a specific Python version, which indicates that if you use it with a different version of Python you're likely to run into troubles.
If that macro fails, shouldn't there be any error messages somewhere?
Well, yeah, should being the key word here. You do realize that we're talking about the guys who keep failing to get some simple includes right...? Just sayin... :mrgreen:
How would I debug that?
That is a very good question...

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Arch compile error in Python[ServerFramework|AI].cpp

#13 Post by Dilvish »

Chriss wrote:If that macro fails, shouldn't there be any error messages somewhere? How would I debug that?
Well, I'm just getting this info via my IDE (KDevelop)-- when I mouseover line 98 "initfo_logger();" it tells me it is defined at line 36, and then when I mouseover the "BOOST_PYTHON_MODULE" macro at line 36, it tells me it is defined in /usr/include/boost/python/module.hpp at line 11, which is simply

Code: Select all

# define BOOST_PYTHON_MODULE BOOST_PYTHON_MODULE_INIT
and "BOOST_PYTHON_MODULE_INIT" itself is defined in /usr/include/boost/python/module_init.hpp. When I mouseover line 36 my IDE also seems to be telling me that the name should be winding up as "init_module_fo_logger()" instead of the "initfo_logger()" that appears at line 98, and that fits with what I see in the boost files-- maybe you want to try changing line 98 to refer to "init_module_fo_logger()". I currently don't really see why "initfo_logger()" is working for me, lol.

As for error messages, you sure you're not just overlooking something in all the boost warning spam we get?
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Arch compile error in Python[ServerFramework|AI].cpp

#14 Post by Chriss »

Vezzra wrote:
Chriss wrote:Hm... I'd really like to not have an extra boost build... Also since the Arch Package would need that too. It seems odd because older boost packages have the same issues. I don't think that python3 stuff is all that new in arch, though I do not know for sure.
That may all be, but fact is that boost python has to be built/linked against a specific Python version, which indicates that if you use it with a different version of Python you're likely to run into troubles.
As far as I can tell, arch's boost has been linked against python2 AND python3 since at least 1.55. That's a long time ago, and I had lots of successful builds in between...

Code: Select all

echo "using python : 3.4 : /usr/bin/python3 : /usr/include/python3.4m : /usr/lib ;" >> ./tools/build/v2/user-config.jam
I'll give the other function name a try and see about compiling boost myself...
Attached patches are released under GPL 2.0 or later.

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Arch compile error in Python[ServerFramework|AI].cpp

#15 Post by Chriss »

I found this: http://www.gamedev.net/topic/661109-cma ... python-27/

And changed the root CMakeLists.txt:

Code: Select all

[chris@desk freeorion.git]$ git diff CMakeLists.txt 
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 46bfa7a..38de16b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -76,7 +76,7 @@ else ()
     message(FATAL_ERROR "Unknown platform type! Your setup is not a supported platform for FreeOrion.")
 endif ()
 
-find_package(PythonInterp REQUIRED)
+find_package(PythonInterp 2 REQUIRED)
 
 # To run the version generation every compile we need to deferr the
 # execution to a separate target and the existing python command
Which seems to fix the issue. So yes, it seems to be an issue with the linked python, but it is not necessary to reinstall boost with different python locations. One can specify the version and get 2 instead of 3. I am not sure if that works in all cases, but maybe someone else (adrian_brother?) can shed some light on that. One needs to clear the build directory (I think), and there were some hints that specifying 2 means "at least 2", which could in theory mean 3, also.

Well, enough for today from me.
Attached patches are released under GPL 2.0 or later.

Post Reply