Potential replacement of FOCS with Python

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Message
Author
Ophiuchus
Programmer
Posts: 3427
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Potential replacement of FOCS with Python

#31 Post by Ophiuchus »

Geoff the Medio wrote: Sun May 31, 2020 2:41 pm
Ophiuchus wrote: Sun May 31, 2020 2:11 pmI am interested in creating the c++ building types/species... via python. So it seems that is currently not possible.

Does any of you pythonistas have experience with creating c data structures from python? Or would i need to provide an api like we have and expose that?
Even if there was a way to create the C++ data structures in Python, similar to how they are created when parsing FOCS files, you'd need to then pass them into the server and clients, either sending them from server to clients or ensuring they are created the same way on both. You'd need some sort of API exposed to do that regardless.
Ah i thought it could be possible to invoke the python interpreter from inside the c++ program. Maybe having an API makes more sense regardless.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#32 Post by o01eg »

Ophiuchus wrote: Sun May 31, 2020 6:57 pm
Geoff the Medio wrote: Sun May 31, 2020 2:41 pm
Ophiuchus wrote: Sun May 31, 2020 2:11 pmI am interested in creating the c++ building types/species... via python. So it seems that is currently not possible.

Does any of you pythonistas have experience with creating c data structures from python? Or would i need to provide an api like we have and expose that?
Even if there was a way to create the C++ data structures in Python, similar to how they are created when parsing FOCS files, you'd need to then pass them into the server and clients, either sending them from server to clients or ensuring they are created the same way on both. You'd need some sort of API exposed to do that regardless.
Ah i thought it could be possible to invoke the python interpreter from inside the c++ program. Maybe having an API makes more sense regardless.
It is done with boost-python
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13586
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Potential replacement of FOCS with Python

#33 Post by Geoff the Medio »

Ophiuchus wrote: Sun May 31, 2020 6:57 pmAh i thought it could be possible to invoke the python interpreter from inside the c++ program. Maybe having an API makes more sense regardless.
It is possible for C++ to execute Python scripts. But for the Python code to access gamestate or scripted content, or to communicate back into C++ code directly at runtime, it needs an API. Such an API exists for turn events and order issuing, but not for content generation.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Potential replacement of FOCS with Python

#34 Post by adrian_broher »

Geoff the Medio wrote: Sun May 31, 2020 10:14 am -How can you use Python right now to define stuff like building types or species? You can't. Such a system doesn't exist.
The Python bindings to game entity descriptions like ShipHull, ShipPart, BuildingType exist, but are not complete, mostly read only and the binding is configured to not allow instantiation of those game entity descriptions. There is also no API to register those entity descriptions into the currently running game.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

ThinkSome
Psionic Snowflake
Posts: 460
Joined: Sun Mar 29, 2020 11:13 pm

Re: Potential replacement of FOCS with Python

#35 Post by ThinkSome »

Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.

Ophiuchus
Programmer
Posts: 3427
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Potential replacement of FOCS with Python

#36 Post by Ophiuchus »

ThinkSome wrote: Fri Jun 05, 2020 10:32 am Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.
Sure, go for it :lol:
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
em3
Vacuum Dragon
Posts: 630
Joined: Sun Sep 25, 2011 2:51 pm

Re: Potential replacement of FOCS with Python

#37 Post by em3 »

ThinkSome wrote: Fri Jun 05, 2020 10:32 am Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.
Why would you want to cross-compile? The code could be compiled as part of installation process or on the first run.
https://github.com/mmoderau
[...] for Man has earned his right to hold this planet against all comers, by virtue of occasionally producing someone totally batshit insane. - Randall Munroe, title text to xkcd #556

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#38 Post by o01eg »

ThinkSome wrote: Fri Jun 05, 2020 10:32 am Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.
I cross-compiled it for Android and even cross-compiled boost with python support.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

ThinkSome
Psionic Snowflake
Posts: 460
Joined: Sun Mar 29, 2020 11:13 pm

Re: Potential replacement of FOCS with Python

#39 Post by ThinkSome »

o01eg wrote: Fri Jun 05, 2020 1:46 pm
ThinkSome wrote: Fri Jun 05, 2020 10:32 am Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.
I cross-compiled it for Android and even cross-compiled boost with python support.
Then you are welcome to join the effort at MXE. Multiple people so far have told me that Python is hard to cross compile and that being the reason why MXE lacks Python support. The only Python-for-MXE project I had found, has heavily patched python 3.3 and even that is broken. It took me some time (and patching) to cross compile boost-python with it and the end result is not very usable for building Freeorion.

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#40 Post by o01eg »

ThinkSome wrote: Fri Jun 05, 2020 2:59 pm Then you are welcome to join the effort at MXE. Multiple people so far have told me that Python is hard to cross compile and that being the reason why MXE lacks Python support. The only Python-for-MXE project I had found, has heavily patched python 3.3 and even that is broken. It took me some time (and patching) to cross compile boost-python with it and the end result is not very usable for building Freeorion.
Could you try this project? https://github.com/python-cmake-buildsy ... ildsystem/
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

User avatar
Oberlus
Cosmic Dragon
Posts: 5704
Joined: Mon Apr 10, 2017 4:25 pm

Re: Potential replacement of FOCS with Python

#41 Post by Oberlus »

Don't feed the troll (even if it is not malicious), you will only get to pollute the thread.

ThinkSome
Psionic Snowflake
Posts: 460
Joined: Sun Mar 29, 2020 11:13 pm

Re: Potential replacement of FOCS with Python

#42 Post by ThinkSome »

o01eg wrote: Fri Jun 05, 2020 4:04 pm Could you try this project? https://github.com/python-cmake-buildsy ... ildsystem/
I will, thank you.

User avatar
Oberlus
Cosmic Dragon
Posts: 5704
Joined: Mon Apr 10, 2017 4:25 pm

Re: Potential replacement of FOCS with Python

#43 Post by Oberlus »

Vezzra wrote: Thu Oct 29, 2020 1:56 pm
FOCS replacements
That question came up because of a small exchange between me and o01eg in the comment section of one of the PRs on github. I mentioned that there is the idea of replacing FOCS with Python (to get rid of our custom built parser, which depends on the Boost parser lib, which has it's own set of issues). o01eg pointed out that replacing our parser with Python will cause issues with background parsing because of Pythons Global Interpreter Lock.

However, considering that replacing the parser will be a quite huge undertaking, which currently no one is willing to tackle anyway, this is something in the far future. So we decided that there is no need to decide anything regarding this right now. For the time being FOCS and our own parser is going to stay.
I have no idea what Global Interpreter Lock (GIL) was until now. Out of curiosity I searched a bit:
If the GIL is causing you problems, here a few approaches you can try:

Multi-processing vs multi-threading: The most popular way is to use a multi-processing approach where you use multiple processes instead of threads. Each Python process gets its own Python interpreter and memory space so the GIL won’t be a problem. Python has a multiprocessing module which lets us create processes easily [...]
https://realpython.com/python-gil/

Would that be feasible for FO?

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#44 Post by o01eg »

I suppose we can partially mitigate GIL issue. We can parse with Python in a single thread but for server and AI it will be main thread but client could use background thread.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

Ophiuchus
Programmer
Posts: 3427
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Potential replacement of FOCS with Python

#45 Post by Ophiuchus »

o01eg wrote: Thu Oct 29, 2020 3:49 pm I suppose we can partially mitigate GIL issue. We can parse with Python in a single thread but for server and AI it will be main thread but client could use background thread.
I guess we could also greatly speed parsing up if we serialize the parsing result and reuse that if there were no content changes.
If that is so, we could run the parser exactly once on first run and in its own process.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

Post Reply