Potential replacement of FOCS with Python

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Message
Author
User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13586
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Potential replacement of FOCS with Python

#61 Post by Geoff the Medio »

Vezzra wrote: Sun Nov 01, 2020 12:21 pmAnd the idea to replace FOCS wih Python is not to change the FOCS stuff to Python syntax and then again use some tools to parse that stuff ourselves, but to actually execute those Python scripts and construct all the condition and effects objects via Python...
Not sure what you mean by "parse that stuff ourselves". o01eg appears to be implementing how I understood the FOCS -> Python plan to work, which is to use the Python code parser to generate a Python data structure and convert that to FreeOrion content data:
o01eg wrote: Sun Nov 01, 2020 1:18 pm...walk Python AST to convert it into FOCS value refs tree...
Once one has a data structure in memory, there's no addition "parsing" required to generate FreeOrion internal ValueRefs, Effects, Conditions, Building or Tech definitions, etc. But doing this parsing and data structure conversion should not involve executing Python scripts.

User avatar
Vezzra
Release Manager, Design
Posts: 6090
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Potential replacement of FOCS with Python

#62 Post by Vezzra »

o01eg wrote: Sun Nov 01, 2020 1:18 pmAlso if user doesn't host the game he doesn't execute all those python code and doesn't affected by its vulnerability.
As that's the (rare) exception (the vast majority plays single player), this isn't really relevant IMO.
We should either import script and call func in Python's interpreter
Well, that's what I thought it's supposed to be. You expose the functions which create all the condition, effect, valueref, etc. objects to Python and call them from there.

User avatar
Vezzra
Release Manager, Design
Posts: 6090
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Potential replacement of FOCS with Python

#63 Post by Vezzra »

Geoff the Medio wrote: Sun Nov 01, 2020 2:01 pmo01eg appears to be implementing how I understood the FOCS -> Python plan to work, which is to use the Python code parser to generate a Python data structure and convert that to FreeOrion content data
I see... I never thought of such an approach. As I said in my post above to o01eg, I always thought the approach would be to expose all the necessary functions to create the condition, effects, valueref etc. objects to Python and create them by executing Python scripts.

But if that other approach is simpler/better, sure, why not. As long as it can be done without adding more dependencies...

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13586
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Potential replacement of FOCS with Python

#64 Post by Geoff the Medio »

Vezzra wrote: Sun Nov 01, 2020 2:10 pmI never thought of such an approach.
See the first post of this thread.

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#65 Post by o01eg »

Vezzra wrote: Sun Nov 01, 2020 2:10 pm
Geoff the Medio wrote: Sun Nov 01, 2020 2:01 pmo01eg appears to be implementing how I understood the FOCS -> Python plan to work, which is to use the Python code parser to generate a Python data structure and convert that to FreeOrion content data
I see... I never thought of such an approach. As I said in my post above to o01eg, I always thought the approach would be to expose all the necessary functions to create the condition, effects, valueref etc. objects to Python and create them by executing Python scripts.

But if that other approach is simpler/better, sure, why not. As long as it can be done without adding more dependencies...
Hard to say what is simpler because we don't have either wrappers for structures or converters from Python AST.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

Ophiuchus
Programmer
Posts: 3427
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Potential replacement of FOCS with Python

#66 Post by Ophiuchus »

Geoff the Medio wrote: Sun Nov 01, 2020 2:01 pm
o01eg wrote: Sun Nov 01, 2020 1:18 pm...walk Python AST to convert it into FOCS value refs tree...
Once one has a data structure in memory, there's no addition "parsing" required to generate FreeOrion internal ValueRefs, Effects, Conditions, Building or Tech definitions, etc. But doing this parsing and data structure conversion should not involve executing Python scripts.
What I would like to have is some kind of python preprocessing/metaprogramming available. Like generating the species colony buildings. But the of course one could preprocess outside - run a generator script which generates content files (like it is done currently for buildings) and checkin the generated content in git/bundle it with the plugin.

If the content is specified in (simplified) python syntax, one can simply import it in a python script (and use the structures or AST to generate the simplified python content).
Such a preprocessing step could also be the way to implement focs macro expansion, although i would rather like to have that as part of simplified python (assignment to constants).
Vezzra wrote: Sun Nov 01, 2020 2:10 pm
Geoff the Medio wrote: Sun Nov 01, 2020 2:01 pmo01eg appears to be implementing how I understood the FOCS -> Python plan to work, which is to use the Python code parser to generate a Python data structure and convert that to FreeOrion content data
I see... I never thought of such an approach. As I said in my post above to o01eg, I always thought the approach would be to expose all the necessary functions to create the condition, effects, valueref etc. objects to Python and create them by executing Python scripts.
Well that is the conversion part.
o01eg wrote: Sun Nov 01, 2020 6:51 am There nice thread I've just read about it https://stackoverflow.com/questions/306 ... ure-python and it looks like there bunch of holes in tries to execute code.
As thinksome said, real sandboxing of complete python will be easier in lua (or e.g. when using jython). But we do not need that.

YAML is only the import format. And breaking out of a python parser is probably not much easier than breaking out of a yaml parser. Also I do think yaml is not a great match for valuerefs etc.

Doing content definition in simplified python I think using AST we could whitelist a very few operations and keep it secure. But depending on how we want valuerefs and conditions to look, executing the definition might not make any sense.

While simple data structures like the gamerules are probably a good point to start dabbling with AST and conversion, it completely sidesteps the difficult parts.

So how would the conditions/valuerefs look like? Like python functions? Like python operations?

Code: Select all

# straightforward focs conditions; emulating value refs via property access in python, this is not really working python. source would be undefined, execution timing is unclear:
scope =  And( Ship(), OwnedBy(source.empire) )

# scope function returns predicates:
def scope():
    return And( Ship(), OwnedBy(Source().EmpireID()) )

# scope function evaluates parameters, using python operators, lots of mentioning rootCandidate which is usually implicit:
def scope(source: Planet, rootCandidate):
    return rootCandidate isinstance Ship && OwnedBy(rootCandidate, source.empireID)

# builder pattern for predicate, call daisy-chaining
def scope(source: Planet, rootCandidate) {
    return rootCandidate.isShip().ownedBy(source.empireId)
# 
something pythonic using member functions and properties for effects, using type hinting for IDE support:

Code: Select all

def effects(source: Planet, target: Ship) {
    if source.defense > target.structure:
        target.destroy()
    else:
        target.structure -= source.defense 
}  
ok, thats probably already too long.. ill stop here ;)
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

ThinkSome
Psionic Snowflake
Posts: 460
Joined: Sun Mar 29, 2020 11:13 pm

Re: Potential replacement of FOCS with Python

#67 Post by ThinkSome »

<rant>

dev1: "Hey, I hear that people can read our FOCS! We need to make it more unreadable!"
dev2: "Yeah, let's call FLAK SR_WEAPON_0_1 instead of FLAK, that'll show them!"

</rant>

Ophiuchus
Programmer
Posts: 3427
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Potential replacement of FOCS with Python

#68 Post by Ophiuchus »

ThinkSome wrote: Fri Nov 06, 2020 11:58 am </rant>
Off topic. We are talking about FOCS, not about content
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
Vezzra
Release Manager, Design
Posts: 6090
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Potential replacement of FOCS with Python

#69 Post by Vezzra »

Ophiuchus wrote: Sun Nov 01, 2020 5:37 pmWhat I would like to have is some kind of python preprocessing/metaprogramming available. Like generating the species colony buildings.
That's exactly what I thought too. Which is why I thought we were going to have an API exposing all the function calls necessary to construct all the condition, effect, valueref etc. objects, and then have Python scripts which would create all the content by calling those functions.

That would allow to use all the features Python provides in these scripts. Macros, metaprogramming, the species colony buildings are a prime example here could be done very easily. No need to do that anymore:
But the of course one could preprocess outside - run a generator script which generates content files (like it is done currently for buildings) and checkin the generated content in git/bundle it with the plugin.
Wouldn't it be good to get rid of that? If we could do that right in the content scripts themselves?

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#70 Post by o01eg »

Vezzra wrote: Sun Nov 08, 2020 1:25 pm
Ophiuchus wrote: Sun Nov 01, 2020 5:37 pmWhat I would like to have is some kind of python preprocessing/metaprogramming available. Like generating the species colony buildings.
That's exactly what I thought too. Which is why I thought we were going to have an API exposing all the function calls necessary to construct all the condition, effect, valueref etc. objects, and then have Python scripts which would create all the content by calling those functions.
Either way we still need to add python dependency to common library, introduce background parsing thread i.e. merge https://github.com/freeorion/freeorion/pull/3211 except some parse/PythonParser.cpp code.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#71 Post by o01eg »

Draft of interpreter variant for game rules: https://github.com/freeorion/freeorion/pull/3265
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#72 Post by o01eg »

I've started to implements techs parser and implemented "macros" a-like system: https://github.com/freeorion/freeorion/pull/3309

While working on ValueRef parsing I found out that FOCS parser uses known context to determine type of ValueRef<int> for Tech researchcost and ValueRef<double> for Tech researchturns.

But for Python execution we start from leafs of tree.

If I have next python files

Code: Select all

TECH_COST_MULTIPLIER = GameRule(name="RULE_TECH_COST_FACTOR")

Code: Select all

from common.base_prod import TECH_COST_MULTIPLIER

Tech(
    name="LRN_ALGO_ELEGANCE",
    description="LRN_ALGO_ELEGANCE_DESC",
    short_description="RESEARCH_SHORT_DESC",
    category="LEARNING_CATEGORY",
    researchcost=10 * TECH_COST_MULTIPLIER,
    researchturns=3,
    tags=["PEDIA_LEARNING_CATEGORY"],
    unlock=Item(type=Policy, name="PLC_ALGORITHMIC_RESEARCH"),
    graphic="icons/tech/algorithmic_elegance.png")
When I call GameRule function I don't know which type of ValueRef I need to get.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13586
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Potential replacement of FOCS with Python

#73 Post by Geoff the Medio »

o01eg wrote: Thu Mar 04, 2021 9:50 pmWhile working on ValueRef parsing I found out that FOCS parser uses known context to determine type of ValueRef<int> for Tech researchcost and ValueRef<double> for Tech researchturns.
Other way around... turns is a int and cost is a double.
I've started to implements techs parser and implemented "macros" a-like system: https://github.com/freeorion/freeorion/pull/3309

Code: Select all

TECH_COST_MULTIPLIER = GameRule(name="RULE_TECH_COST_FACTOR")
[...]
When I call GameRule function I don't know which type of ValueRef I need to get.
In FOCS, macros are just text substitution, so don't need to be a fully-formed parseable anything before being substituted...

If that's the issue, can you just treat macros as text substitution, and not parse anything in them directly?

Or would it still be a problem if you wrote out, without macros:

Code: Select all

researchcost=10 * GameRule(name="RULE_TECH_COST_FACTOR")
In that case, I guess you could make separate rule-getter calls, like GetDoubleRule, GetIntRule, GetStringRule, etc.

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#74 Post by o01eg »

Geoff the Medio wrote: Thu Mar 04, 2021 11:37 pm In FOCS, macros are just text substitution, so don't need to be a fully-formed parseable anything before being substituted...

If that's the issue, can you just treat macros as text substitution, and not parse anything in them directly?
Python doesn't have macroses so I decided to use python way of using common code. If I introduce some macros string substitution I won't be able to check syntax with PEP8 or execute those files as regular python files.
Geoff the Medio wrote: Thu Mar 04, 2021 11:37 pm In that case, I guess you could make separate rule-getter calls, like GetDoubleRule, GetIntRule, GetStringRule, etc.
Yes, I also thought about building untyped tree and them write types back when assign it to known field but it will be slower than just wrap correctly typed ValueRef while execute python code and unwrap it when assign.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

o01eg
Programmer
Posts: 1998
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#75 Post by o01eg »

Finished minimal techs parser for two techs Algorithmic Elegance and Orbital Habitation.

All other techs aren't converted but it shows how Python parser could looks like for value refs, conditions, and effects.

At least it passed parser tests locally.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-01-30.0dd6806.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

Post Reply