Build procedure is not python3 compatible

Questions, problems and discussion about compiling FreeOrion.
Message
Author
User avatar
adrian_broher
Programmer
Posts: 1072
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Build procedure is not python3 compatible

#16 Post by adrian_broher » Tue May 26, 2015 8:09 am

Vezzra wrote:It apparently insists on using 'sh' instead of 'bash', without including '/usr/local/bin' into PATH.
I see. What does the PATH look like in the first place? Why does XCode use sh? Does it honor /etc/profile? Does /etc/profile not contain a line like `/usr/libexec/path_helper -s`? Executing that application should return something like: "PATH="/usr/bin:/bin:/usr/sbin:/usr/local/bin:/usr/X11/bin"; export PATH;"
Cjkjvfnby wrote:On linux is not (I think it is issue)
There is no issue with relying on dependencies provided by the platform.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Cjkjvfnby
AI Contributor
Posts: 448
Joined: Tue Jun 24, 2014 9:55 pm

Re: Build procedure is not python3 compatible

#17 Post by Cjkjvfnby » Tue May 26, 2015 8:29 am

adrian_broher wrote:
Vezzra wrote:
Cjkjvfnby wrote:On linux is not (I think it is issue)
There is no issue with relying on dependencies provided by the platform.
It can cause strange bugs in AI code.

Pyhton 2.7 is 5 years old, so where is no many platforms with default instllation of other versions.
Python 2.7 support will end in 5 years, so I expect number of systems with default python3 installation will grow. I think we will back to this discussion in couple of years.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
adrian_broher
Programmer
Posts: 1072
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Build procedure is not python3 compatible

#18 Post by adrian_broher » Tue May 26, 2015 8:39 am

Cjkjvfnby wrote:Pyhton 2.7 is 5 years old, so where is no many platforms with default instllation of other versions.
Python 2.7 support will end in 5 years, so I expect number of systems with default python3 installation will grow. I think we will back to this discussion in couple of years.
I'm aware of that. But how does packaging an (in the future) unmaintained version of python fix or even address that issue?
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Build procedure is not python3 compatible

#19 Post by Chriss » Tue May 26, 2015 10:05 pm

My feeling is that currently, both python versions are commonly used and available on most (linux) platforms. Arch and gentoo use 3 as default, more conservative distros use 2 as default. Both distro types should have both python versions available, probably even installed by default. The PEP mentions that Python2 will be supported at least til 2020, so there is no immediate rush to migrate. No harm in it of course, and if it's just the print builtin / function that causes the issue...
Attached patches are released under GPL 2.0 or later.

User avatar
Vezzra
Release Manager, Design
Posts: 4609
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build procedure is not python3 compatible

#20 Post by Vezzra » Wed May 27, 2015 10:15 am

Dilvish wrote:
Cjkjvfnby wrote:I don`t think that build script should have limitation on python version.
Why is that? If our main app requires python2.7, I don't see an issue with the build script also requiring it.
Bascially true, but the build script isn't that complicated, so it shouldn't be too hard trying to make it Python version independent (I hope). It certainly won't hurt, will probably improve maintainability a bit, especially on OSX, where the build script can't use the bundled Python version, but always uses the one installed on the system.

User avatar
Vezzra
Release Manager, Design
Posts: 4609
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build procedure is not python3 compatible

#21 Post by Vezzra » Wed May 27, 2015 10:33 am

adrian_broher wrote:What does the PATH look like in the first place?
On the terminal:

Code: Select all

$ echo $PATH
/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin:/Developer/SDKs/OgreSDK/bin
Output of 'echo $PATH' from within the Xcode build script:

Code: Select all

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin:/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/libexec:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/bin:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/local/bin:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/usr/bin:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/usr/local/bin:/Applications/Xcode.app/Contents/Developer/usr/bin:/Applications/Xcode.app/Contents/Developer/usr/local/bin:/Applications/Xcode.app/Contents/Developer/Tools:/usr/bin:/bin:/usr/sbin:/sbin
Why does XCode use sh?
Only the Apple Powers That Be will know... anyway, irrelevant, as at least recent versions of OSX hard link 'sh' to 'bash' (as I found out). So that's not the problem, it's definitely the differences with PATH.
Does it honor /etc/profile?
Can't say for sure, but apparently not.
Does /etc/profile not contain a line like `/usr/libexec/path_helper -s`?
/etc/profile:

Code: Select all

# System-wide .profile for sh(1)

if [ -x /usr/libexec/path_helper ]; then
	eval `/usr/libexec/path_helper -s`
fi

if [ "${BASH-no}" != "no" ]; then
	[ -r /etc/bashrc ] && . /etc/bashrc
fi
Executing that application should return something like: "PATH="/usr/bin:/bin:/usr/sbin:/usr/local/bin:/usr/X11/bin"; export PATH;"
Output when executing that command:

Code: Select all

PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/opt/local/bin:/opt/local/sbin:/Developer/SDKs/OgreSDK/bin"; export PATH;

User avatar
Cjkjvfnby
AI Contributor
Posts: 448
Joined: Tue Jun 24, 2014 9:55 pm

Re: Build procedure is not python3 compatible

#22 Post by Cjkjvfnby » Wed May 27, 2015 11:27 pm

Vezzra wrote:
Dilvish wrote:
Cjkjvfnby wrote:I don`t think that build script should have limitation on python version.
Why is that? If our main app requires python2.7, I don't see an issue with the build script also requiring it.
Bascially true, but the build script isn't that complicated, so it shouldn't be too hard trying to make it Python version independent (I hope). It certainly won't hurt, will probably improve maintainability a bit, especially on OSX, where the build script can't use the bundled Python version, but always uses the one installed on the system.
In python you can wrap value to parenthesis and get same value (to create tuple with one element you need coma.)

print (a) is same as print a in python2. Python 3 parse it as call of function print()
https://raw.githubusercontent.com/Cjkjv ... sioncpp.py
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12249
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Build procedure is not python3 compatible

#23 Post by Geoff the Medio » Wed May 27, 2015 11:39 pm

Is there anything preventing all the Python code associated with FreeOrion from being made compatible with Python3 (without breaking Python2 compatibility)? What besides writing print(x) instead of print x would need substantial changes?

User avatar
Cjkjvfnby
AI Contributor
Posts: 448
Joined: Tue Jun 24, 2014 9:55 pm

Re: Build procedure is not python3 compatible

#24 Post by Cjkjvfnby » Thu May 28, 2015 7:26 am

Geoff the Medio wrote:Is there anything preventing all the Python code associated with FreeOrion from being made compatible with Python3 (without breaking Python2 compatibility)? What besides writing print(x) instead of print x would need substantial changes?
All integer division must be checked
Functions like zip and map in python 3 returns an iterator not list.
May be reorganize some imports.

I dont want to support both versions at one time. It will require additional testing.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
adrian_broher
Programmer
Posts: 1072
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Build procedure is not python3 compatible

#25 Post by adrian_broher » Thu May 28, 2015 7:50 am

Geoff the Medio wrote:Is there anything preventing all the Python code associated with FreeOrion from being made compatible with Python3?
The SDKs and the application design. You can't run the python iinterpreter in mixed mode, you can't provide two different python interpreters at the same time.
without breaking Python2 compatibility?
See above.
What besides writing print(x) instead of print x would need substantial changes?
Complete list: https://docs.python.org/3/whatsnew/3.0.html

Things that are probably relevant:
  • print function instead of print keyword. (Cjkjvfnby suggestion is hardly an equivalent, considering the additional parameters possible).
  • The comparing `<>` was removed in favour of `!=`.
  • relative imports are only allowed as `from . import package`.
  • Exceptions are cached like `except Type as variable` instead of `except Type, variable`.
  • Comparing different types with `<=`, `==`, `>=` will lead to TypeError (`1 <= ''` or `1 == None` for example).
  • `1/2` returns a float. `1//2` truncates the result to int.
  • string are by default unicode. They store code points. If you need the write or read the string 'somewhere external' you need to de- or encode it.
This cheat sheed should help: http://python-future.org/compatible_idioms.html
Also this online book explains stuff in detail: http://python3porting.com/
Last edited by adrian_broher on Thu May 28, 2015 3:10 pm, edited 1 time in total.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 4609
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build procedure is not python3 compatible

#26 Post by Vezzra » Thu May 28, 2015 2:45 pm

Geoff the Medio wrote:Is there anything preventing all the Python code associated with FreeOrion from being made compatible with Python3 (without breaking Python2 compatibility)?
Marcel already pointed out several things, making Python code compatible to both Python 2&3 certainly isn't a trivial thing. I'd expect that to require considerable effort, but even more to keep the Python code 2&3 compatible as we continue developing. That sounds messy... and I don't think I want to go there.

Doing the switch to Python 3 will be trouble enough once we decide to do that ;)

User avatar
Vezzra
Release Manager, Design
Posts: 4609
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build procedure is not python3 compatible

#27 Post by Vezzra » Thu May 28, 2015 2:53 pm

adrian_broher wrote:
Geoff the Medio wrote:Is there anything preventing all the Python code associated with FreeOrion from being made compatible with Python3?
The SDKs and the application design. You can't run the python iinterpreter in mixed mode, you can't provide two different python iinterpreters at the same time.
Well, that wouldn't be necessary, Geoff merely suggests making the Python code compatible with both Python 2&3, so that it can run with either. So regardless which Python version you bundle with the app, the code will work.

It's the setup of development/test environment that would give me serious headache if we seriously considered doing that. You'd need to maintain a setup where you're able to test your code with both Python versions. And I expect Python coding to be more tedious if you constantly try to keep everything 2&3 compatible.

Ugh. Doesn't sound like a good idea.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12249
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Build procedure is not python3 compatible

#28 Post by Geoff the Medio » Thu May 28, 2015 11:37 pm

It seems likely that it will be necessary to transition to Python3 at some point. Keeping compatible with both during the transition seems like it might be useful.

User avatar
Vezzra
Release Manager, Design
Posts: 4609
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build procedure is not python3 compatible

#29 Post by Vezzra » Fri May 29, 2015 1:16 pm

Geoff the Medio wrote:It seems likely that it will be necessary to transition to Python3 at some point. Keeping compatible with both during the transition seems like it might be useful.
Why/how? I'd suggest to do this like we did the switch from Ogre to SDL. Make a migration branch and new experimental SDKs for Python 3, do all the migration work on that branch. Once finished and sufficiently tested, merge the migration branch into master and make the experimental SDKs the current ones.

I'd expect trying to make our Python scripts 2&3 compatible to be considerable more difficult than just switching.

What benefit do you see in keeping compatible with 2&3 during the transition?

User avatar
Dilvish
AI Lead, Programmer
Posts: 4685
Joined: Sat Sep 22, 2012 6:25 pm

Re: Build procedure is not python3 compatible

#30 Post by Dilvish » Fri May 29, 2015 4:55 pm

Vezzra wrote:I'd expect trying to make our Python scripts 2&3 compatible to be considerable more difficult than just switching.
Python 2/3 Porting Guide wrote:Over the years the Python community has discovered that the easiest way to support both Python 2 and 3 in parallel is to write Python code that works in either version. While this might sound counter-intuitive at first, it actually is not difficult and typically only requires following some select (non-idiomatic) practices and using some key projects to help make bridging between Python 2 and 3 easier.
https://docs.python.org/3.3/howto/pypor ... n-2-3-code

I don't really think it would be that much more difficult, particularly since we already require python 2.7. Python 2.7 has a number of features meant to smooth the transition, including a "-3" option to explicitly test for python 3 compatibility, and the advice will be for us to try to start writing compatible code now. From my initial reading it sounds like the biggest nuisance will be dealing with string literals. I haven't tried out the 2to3 converter, but I get the impression that for our python 2.7 code it will do a fair bit for us.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

Post Reply