Page 3 of 5

Re: Potential replacement of FOCS with Python

Posted: Sun May 31, 2020 6:57 pm
by Ophiuchus
Geoff the Medio wrote: Sun May 31, 2020 2:41 pm
Ophiuchus wrote: Sun May 31, 2020 2:11 pmI am interested in creating the c++ building types/species... via python. So it seems that is currently not possible.

Does any of you pythonistas have experience with creating c data structures from python? Or would i need to provide an api like we have and expose that?
Even if there was a way to create the C++ data structures in Python, similar to how they are created when parsing FOCS files, you'd need to then pass them into the server and clients, either sending them from server to clients or ensuring they are created the same way on both. You'd need some sort of API exposed to do that regardless.
Ah i thought it could be possible to invoke the python interpreter from inside the c++ program. Maybe having an API makes more sense regardless.

Re: Potential replacement of FOCS with Python

Posted: Sun May 31, 2020 6:59 pm
by o01eg
Ophiuchus wrote: Sun May 31, 2020 6:57 pm
Geoff the Medio wrote: Sun May 31, 2020 2:41 pm
Ophiuchus wrote: Sun May 31, 2020 2:11 pmI am interested in creating the c++ building types/species... via python. So it seems that is currently not possible.

Does any of you pythonistas have experience with creating c data structures from python? Or would i need to provide an api like we have and expose that?
Even if there was a way to create the C++ data structures in Python, similar to how they are created when parsing FOCS files, you'd need to then pass them into the server and clients, either sending them from server to clients or ensuring they are created the same way on both. You'd need some sort of API exposed to do that regardless.
Ah i thought it could be possible to invoke the python interpreter from inside the c++ program. Maybe having an API makes more sense regardless.
It is done with boost-python

Re: Potential replacement of FOCS with Python

Posted: Sun May 31, 2020 7:50 pm
by Geoff the Medio
Ophiuchus wrote: Sun May 31, 2020 6:57 pmAh i thought it could be possible to invoke the python interpreter from inside the c++ program. Maybe having an API makes more sense regardless.
It is possible for C++ to execute Python scripts. But for the Python code to access gamestate or scripted content, or to communicate back into C++ code directly at runtime, it needs an API. Such an API exists for turn events and order issuing, but not for content generation.

Re: Potential replacement of FOCS with Python

Posted: Thu Jun 04, 2020 7:59 pm
by adrian_broher
Geoff the Medio wrote: Sun May 31, 2020 10:14 am -How can you use Python right now to define stuff like building types or species? You can't. Such a system doesn't exist.
The Python bindings to game entity descriptions like ShipHull, ShipPart, BuildingType exist, but are not complete, mostly read only and the binding is configured to not allow instantiation of those game entity descriptions. There is also no API to register those entity descriptions into the currently running game.

Re: Potential replacement of FOCS with Python

Posted: Fri Jun 05, 2020 10:32 am
by ThinkSome
Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.

Re: Potential replacement of FOCS with Python

Posted: Fri Jun 05, 2020 11:06 am
by Ophiuchus
ThinkSome wrote: Fri Jun 05, 2020 10:32 am Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.
Sure, go for it :lol:

Re: Potential replacement of FOCS with Python

Posted: Fri Jun 05, 2020 12:17 pm
by em3
ThinkSome wrote: Fri Jun 05, 2020 10:32 am Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.
Why would you want to cross-compile? The code could be compiled as part of installation process or on the first run.

Re: Potential replacement of FOCS with Python

Posted: Fri Jun 05, 2020 1:46 pm
by o01eg
ThinkSome wrote: Fri Jun 05, 2020 10:32 am Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.
I cross-compiled it for Android and even cross-compiled boost with python support.

Re: Potential replacement of FOCS with Python

Posted: Fri Jun 05, 2020 2:59 pm
by ThinkSome
o01eg wrote: Fri Jun 05, 2020 1:46 pm
ThinkSome wrote: Fri Jun 05, 2020 10:32 am Python is hard to cross-compile, I'd rather see FOCS replaced with Lua.
I cross-compiled it for Android and even cross-compiled boost with python support.
Then you are welcome to join the effort at MXE. Multiple people so far have told me that Python is hard to cross compile and that being the reason why MXE lacks Python support. The only Python-for-MXE project I had found, has heavily patched python 3.3 and even that is broken. It took me some time (and patching) to cross compile boost-python with it and the end result is not very usable for building Freeorion.

Re: Potential replacement of FOCS with Python

Posted: Fri Jun 05, 2020 4:04 pm
by o01eg
ThinkSome wrote: Fri Jun 05, 2020 2:59 pm Then you are welcome to join the effort at MXE. Multiple people so far have told me that Python is hard to cross compile and that being the reason why MXE lacks Python support. The only Python-for-MXE project I had found, has heavily patched python 3.3 and even that is broken. It took me some time (and patching) to cross compile boost-python with it and the end result is not very usable for building Freeorion.
Could you try this project? https://github.com/python-cmake-buildsy ... ildsystem/

Re: Potential replacement of FOCS with Python

Posted: Fri Jun 05, 2020 4:24 pm
by Oberlus
Don't feed the troll (even if it is not malicious), you will only get to pollute the thread.

Re: Potential replacement of FOCS with Python

Posted: Fri Jun 05, 2020 4:48 pm
by ThinkSome
o01eg wrote: Fri Jun 05, 2020 4:04 pm Could you try this project? https://github.com/python-cmake-buildsy ... ildsystem/
I will, thank you.

Re: Potential replacement of FOCS with Python

Posted: Thu Oct 29, 2020 3:42 pm
by Oberlus
Vezzra wrote: Thu Oct 29, 2020 1:56 pm
FOCS replacements
That question came up because of a small exchange between me and o01eg in the comment section of one of the PRs on github. I mentioned that there is the idea of replacing FOCS with Python (to get rid of our custom built parser, which depends on the Boost parser lib, which has it's own set of issues). o01eg pointed out that replacing our parser with Python will cause issues with background parsing because of Pythons Global Interpreter Lock.

However, considering that replacing the parser will be a quite huge undertaking, which currently no one is willing to tackle anyway, this is something in the far future. So we decided that there is no need to decide anything regarding this right now. For the time being FOCS and our own parser is going to stay.
I have no idea what Global Interpreter Lock (GIL) was until now. Out of curiosity I searched a bit:
If the GIL is causing you problems, here a few approaches you can try:

Multi-processing vs multi-threading: The most popular way is to use a multi-processing approach where you use multiple processes instead of threads. Each Python process gets its own Python interpreter and memory space so the GIL won’t be a problem. Python has a multiprocessing module which lets us create processes easily [...]
https://realpython.com/python-gil/

Would that be feasible for FO?

Re: Potential replacement of FOCS with Python

Posted: Thu Oct 29, 2020 3:49 pm
by o01eg
I suppose we can partially mitigate GIL issue. We can parse with Python in a single thread but for server and AI it will be main thread but client could use background thread.

Re: Potential replacement of FOCS with Python

Posted: Fri Oct 30, 2020 7:18 am
by Ophiuchus
o01eg wrote: Thu Oct 29, 2020 3:49 pm I suppose we can partially mitigate GIL issue. We can parse with Python in a single thread but for server and AI it will be main thread but client could use background thread.
I guess we could also greatly speed parsing up if we serialize the parsing result and reuse that if there were no content changes.
If that is so, we could run the parser exactly once on first run and in its own process.