Build procedure is not python3 compatible

Questions, problems and discussion about compiling FreeOrion.
Message
Author
User avatar
Vezzra
Release Manager, Design
Posts: 4673
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build procedure is not python3 compatible

#31 Post by Vezzra » Fri May 29, 2015 6:09 pm

Dilvish wrote:
Python 2/3 Porting Guide wrote:Over the years the Python community has discovered that the easiest way to support both Python 2 and 3 in parallel is to write Python code that works in either version. While this might sound counter-intuitive at first, it actually is not difficult and typically only requires following some select (non-idiomatic) practices and using some key projects to help make bridging between Python 2 and 3 easier.
https://docs.python.org/3.3/howto/pypor ... n-2-3-code
I stand corrected :D

Apo
Space Squid
Posts: 89
Joined: Fri Apr 19, 2013 4:10 pm

Re: Build procedure is not python3 compatible

#32 Post by Apo » Fri May 29, 2015 7:56 pm

Hi,

there is also 2to3 which might help you with porting your Python2 code to Python3.

https://docs.python.org/2/library/2to3.html

http://www.diveintopython3.net/porting- ... -2to3.html

Debian supports Python2 and Python3 and Python 2.7 is still supported upstream. However there will be no Python 2.8 and Python 2 is scheduled for EOL in 2020.
https://hg.python.org/peps/rev/76d43e52d978

It is not super urgent but of course it would be great to move to Python 3. :)

P.S.: But please don't bundle Python with FreeOrion. That's a No-No for Linux.

User avatar
Dilvish
AI Lead, Programmer
Posts: 4720
Joined: Sat Sep 22, 2012 6:25 pm

Re: Build procedure is not python3 compatible

#33 Post by Dilvish » Fri May 29, 2015 8:09 pm

Reading up a little more, though, I'm getting a bit concerned that boost might substantially complicate our 2/3 compatibility goals. It seems that boost has been a bit sluggish in supporting Python3, and it appears that the most recent boost-python library is only compatible with python 2.x and python 3.4+, but *not* python 3.1.x, 3.2.x or 3.3.x. I've also seen a number of references (possibly stale, but not clearly so) to ongoing boost patches and difficulty with the conversion of std::string, which gets more complicated for python 3, going either to python unicode or to python3 'bytes'. That's already a nontrivial part of python 2to3 upgrading anyway, and we pass a great many strings, and containers of strings, around between C++ and python, so this is a pertinent issue for us.

So, while it probably does still make sense to start writing our python code to be as python3 compatible as possible, it could also be that we might want to let boost support for python3 gestate a bit more before before we concern ourselves too much with full support for python3. If anyone turns up any kind of guide about updating boost-python interfaces for python3 please post about it here.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

Apo
Space Squid
Posts: 89
Joined: Fri Apr 19, 2013 4:10 pm

Re: Build procedure is not python3 compatible

#34 Post by Apo » Fri May 29, 2015 8:41 pm

From Debian's point of view there shouldn't be any problem because Python 3.4 is fully supported and since Debian is one of the more conservative Linux distributions, I assume that's also not a problem for Arch Linux, Gentoo, Fedora and Co.

User avatar
Cjkjvfnby
AI Contributor
Posts: 452
Joined: Tue Jun 24, 2014 9:55 pm

Re: Build procedure is not python3 compatible

#35 Post by Cjkjvfnby » Fri May 29, 2015 9:56 pm

Apo wrote:P.S.: But please don't bundle Python with FreeOrion. That's a No-No for Linux.
Install an play is not Linux way, users will install all missed dependencies themself :)

python 2.7 is 5 years old, so there is almost everyone have 2.7 by default.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Build procedure is not python3 compatible

#36 Post by Chriss » Sun May 31, 2015 8:32 pm

If Debian has 3.4, then Ubuntu should have it (soon). The main blocker I think is Boost. If boost is not python3 compatible, I don't think there's a big point in forcing a conversion. Of course, if the code is transitioned to the mentioned "support both" style, the transition will probably be easier, there's no harm in it, and if it does not hinder your programming you'll probably get some experience with that... Not for me to say though.

But: maybe we can differentiate between the game code, and the build code? Game code is linked and executed against / with 2.7, which can be controlled with cmake and similar. The Build code like the script which prompted this thread is independent of that, and of boost. Since it is not necessarily clear what python version executes this code (depends on other things than cmake), it may be a good idea (and probably a relatively small effort) to support both 2 and 3 with that part of the codebase?
Attached patches are released under GPL 2.0 or later.

Apo
Space Squid
Posts: 89
Joined: Fri Apr 19, 2013 4:10 pm

Re: Build procedure is not python3 compatible

#37 Post by Apo » Mon Jun 01, 2015 8:09 pm

Chriss wrote:The main blocker I think is Boost. If boost is not python3 compatible, I don't think there's a big point in forcing a conversion.
I can't imagine that Boost is Python3 incompatible.

https://packages.debian.org/unstable/li ... on1.55-dev

Debian's boost-python package claims all versions of Python in Debian are supported, so I think Boost won't be the blocker.

User avatar
Dilvish
AI Lead, Programmer
Posts: 4720
Joined: Sat Sep 22, 2012 6:25 pm

Re: Build procedure is not python3 compatible

#38 Post by Dilvish » Mon Jun 01, 2015 10:58 pm

Apo wrote:
Chriss wrote:The main blocker I think is Boost. If boost is not python3 compatible...
I can't imagine that Boost is Python3 incompatible... Debian's boost-python package claims all versions of Python in Debian are supported, so I think Boost won't be the blocker.
We do need some more info, but it seems this line of the conversation keeps circling back on itself and clouding itself. When I had indicated above that it seemed boost was being a little sluggish with python3, I was mostly just trying to convey that their python3 support has probably not been exercised a whole lot yet, which just adds to some uncertainty with it. It seems reasonably likely that boost can do what we want, but it seems to me there are some significant questions outstanding.

So, adding a bit to the info above, also with some more links:
Our current info indicates that boost-python will be supporting python 2.7 and (perhaps with with some problems) python 3.4+. More specifically, the 1.58 boost python build docs say python 2.2 or newer, though their footnotes say this is only tested for python 2.4+). The boost python build docs don't actually specifically mention python 3 in any way, they simply mention the manner to specify one or more python versions that can be built against. From what Apo has said before I think his recent link indicates debian boost python support for 2.7 & 3.4. But our experience with boost (and various threads I have found) indicates there might easily be patches necessary.

Questions that appear to me to be outstanding:
  • Do we need to do anything special to have our boost python interface compatible with both python 2.7 and python 3?This thread (from a couple years ago, but I can't find anything on this point more current) indicates that when you build boost against both python2.7 and python3, it will name the python3 library libboost_python3.so. I suppose it will be easy enough to tell cmake which one to use; on Windows and MacOSX, would we have a different project for python 2.7 and for python3? I had been concerned about different signatures for things, but apparently that is pretty limited and it seems it shouldn't be much of an issue for us. I had been concerned about what changes our boost interface might need because of the python3 change of string to unicode, but now it's seeming to me like that probably won't be a significant problem.
  • Is the boost python 3.x support really just for python3.4+? That's what this link from the boost support forums about a month ago indicates, but I haven't been able to turn up any info more formal. The boost forum link above from early May indicates they thought they had a fix to allow compiling with python 3 < 3.4, but that same thread indicates some problems even for python 3.4.3, with what appears to be a probable fix, but it wasn't totally clear to me. And the fix involved patching boost itself, which we've been keen to avoid.
  • If boost python doesn't support python 3.1-3.3, do we want to spend much time yet worrying about python3?All the newer Linux releases seem to be supporting 3.4, but and I'm sure it's available for Windows and MacOSX, but it's still fairly recent. There is still long term support for many Linux distros with python 3 versions less than 3.4 though. Debian Wheezy is the current 'oldstable', not yet even on what they term Long Term Support, which it will be on from Feb 2016 to May 2018; it appears that the python3 version for Wheezy is 3.2.3. For Ubuntu, it appears python 3.4 was first supported for Trusty (14.04), which was released just a year ago. End of Life for Precise (12.04) is apparently not until April 2017, and for python3 it, like debian wheezy, apparently only supports 3.2.3
I don't know how much that all really matters, but I am left feeling that we don't need to prioritize python3, we can just start trying to work in python3 compatible code slowly over time, for now.

I am thinking my preference would be to start writing python code that is 2/3 compatible
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Vezzra
Release Manager, Design
Posts: 4673
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build procedure is not python3 compatible

#39 Post by Vezzra » Tue Jun 02, 2015 7:01 am

Dilvish wrote:I am thinking my preference would be to start writing python code that is 2/3 compatible
This.

User avatar
adrian_broher
Programmer
Posts: 1072
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Build procedure is not python3 compatible

#40 Post by adrian_broher » Wed Jun 03, 2015 8:25 am

After this thread took a somewhat different direction I would like to point out, that I've fixed the CMake build in a way, that I think is appropriate:
  • Explicit state, that we search for and require a 2.7 compatible interpreter at build time and link the application against 2.7 compatible libraries.
  • We don't rely on the shebang anymore, but invoke the make_versioncpp.py script explicit with the python interpreter executable.
Chriss, could you please check if that fixes your problem? You still need a python 2.7 installation, but CMake should pick up the right interpreter even if `/usr/bin/python` points to python 3.x.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Build procedure is not python3 compatible

#41 Post by Chriss » Thu Jul 23, 2015 9:04 pm

Sorry for the delay. I think there was a workaround even before, but yes, I compiled the git head from today without issues or patches.
Attached patches are released under GPL 2.0 or later.

Post Reply