Potential replacement of FOCS with Python

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Message
Author
User avatar
em3
Vacuum Dragon
Posts: 628
Joined: Sun Sep 25, 2011 2:51 pm

Re: Potential replacement of FOCS with Python

#46 Post by em3 »

I thought Python was not meant to be used as an engine to run the effects, but as a way to construct c++ effects data structures. As such, it would be ran only at the start, right?
https://github.com/macmodrov
[...] for Man has earned his right to hold this planet against all comers, by virtue of occasionally producing someone totally batshit insane. - Randall Munroe, title text to xkcd #556

o01eg
Programmer
Posts: 1007
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#47 Post by o01eg »

I've managed to get AST from file in https://github.com/freeorion/freeorion/pull/3211 so I would like if someone from Python coders could provide Python look of game_rules.focs.txt.

It's better if it can pass CI checks for python.

I've prepared test but the only thing it can do is to validate if file have python syntax.
Gentoo Linux x64, gcc-9.3, boost-1.74.0
Ubuntu Server 18.04 x64, gcc-7.4, boost-1.65.1
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 0.4.10.1.
Donations are welcome: BTC:14XLekD9ifwqLtZX4iteepvbLQNYVG87zK

Ophiuchus
Programmer
Posts: 1849
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Potential replacement of FOCS with Python

#48 Post by Ophiuchus »

o01eg wrote: Fri Oct 30, 2020 2:09 pm..provide Python look of game_rules.focs.txt.
dunno. like this(?):

Code: Select all

class RealGameRule():
    def __init__(self, name, description, category, default, min, max):
	self.name = name
	self.description = description # could be defaulted to name + _DESC here                                                                                                                                                              
        self.category = category # could be stronger typed                                                                                                                                                                                    
        self.default = default
        self.min = min
        self.max = max

BALANCE="BALANCE"

game_rules_list = [
    RealGameRule("RULE_SHIP_HULL_COST_FACTOR",
        description="RULE_SHIP_HULL_COST_FACTOR_DESC",
        category = BALANCE,
        default = 1.0,
        min = 0.1,
	max = 10.0 ),
    RealGameRule("RULE_SHIP_PART_COST_FACTOR",
        description = "RULE_SHIP_PART_COST_FACTOR_DESC",
        category = BALANCE,
        default = 1.0,
        min = 0.1,
        max = 10.0 ),
]


for rule in game_rules_list:
    print("Parsed ", rule.name)
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

o01eg
Programmer
Posts: 1007
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#49 Post by o01eg »

Ophiuchus wrote: Sat Oct 31, 2020 1:25 pm
o01eg wrote: Fri Oct 30, 2020 2:09 pm..provide Python look of game_rules.focs.txt.
dunno. like this(?):

Code: Select all

class RealGameRule():
    def __init__(self, name, description, category, default, min, max):
	self.name = name
	self.description = description # could be defaulted to name + _DESC here                                                                                                                                                              
        self.category = category # could be stronger typed                                                                                                                                                                                    
        self.default = default
        self.min = min
        self.max = max

BALANCE="BALANCE"

game_rules_list = [
    RealGameRule("RULE_SHIP_HULL_COST_FACTOR",
        description="RULE_SHIP_HULL_COST_FACTOR_DESC",
        category = BALANCE,
        default = 1.0,
        min = 0.1,
	max = 10.0 ),
    RealGameRule("RULE_SHIP_PART_COST_FACTOR",
        description = "RULE_SHIP_PART_COST_FACTOR_DESC",
        category = BALANCE,
        default = 1.0,
        min = 0.1,
        max = 10.0 ),
]


for rule in game_rules_list:
    print("Parsed ", rule.name)
Code is only parsed as Python, not executed as Python.
Gentoo Linux x64, gcc-9.3, boost-1.74.0
Ubuntu Server 18.04 x64, gcc-7.4, boost-1.65.1
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 0.4.10.1.
Donations are welcome: BTC:14XLekD9ifwqLtZX4iteepvbLQNYVG87zK

Ophiuchus
Programmer
Posts: 1849
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Potential replacement of FOCS with Python

#50 Post by Ophiuchus »

o01eg wrote: Sat Oct 31, 2020 2:30 pm Code is only parsed as Python, not executed as Python.

Code: Select all

game_rules_list = [
    RealGameRule("RULE_SHIP_HULL_COST_FACTOR",
        description="RULE_SHIP_HULL_COST_FACTOR_DESC",
        category = "BALANCE",
        default = 1.0,
        min = 0.1,
	max = 10.0 ),
    RealGameRule("RULE_SHIP_PART_COST_FACTOR",
        description = "RULE_SHIP_PART_COST_FACTOR_DESC",
        category = "BALANCE",
        default = 1.0,
        min = 0.1,
        max = 10.0 ),
]
I would prefer some execution before using the structure. So one can e.g. define BALANCE (making sure it is everywhere the same). Or use a loop to create multiple definitions.

I am not sure why you want to use AST though (if that is what you want). To save the bytecode conversion/time spent on interpretation?
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

o01eg
Programmer
Posts: 1007
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#51 Post by o01eg »

Ophiuchus wrote: Sat Oct 31, 2020 7:57 pm
o01eg wrote: Sat Oct 31, 2020 2:30 pm Code is only parsed as Python, not executed as Python.

Code: Select all

game_rules_list = [
    RealGameRule("RULE_SHIP_HULL_COST_FACTOR",
        description="RULE_SHIP_HULL_COST_FACTOR_DESC",
        category = "BALANCE",
        default = 1.0,
        min = 0.1,
	max = 10.0 ),
    RealGameRule("RULE_SHIP_PART_COST_FACTOR",
        description = "RULE_SHIP_PART_COST_FACTOR_DESC",
        category = "BALANCE",
        default = 1.0,
        min = 0.1,
        max = 10.0 ),
]
I would prefer some execution before using the structure. So one can e.g. define BALANCE (making sure it is everywhere the same). Or use a loop to create multiple definitions.

I am not sure why you want to use AST though (if that is what you want). To save the bytecode conversion/time spent on interpretation?
Mostly because it could be unsafe to execute python code from untrusted sources and planned modding support could provide amount of untrusted sources.
Gentoo Linux x64, gcc-9.3, boost-1.74.0
Ubuntu Server 18.04 x64, gcc-7.4, boost-1.65.1
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 0.4.10.1.
Donations are welcome: BTC:14XLekD9ifwqLtZX4iteepvbLQNYVG87zK

Ophiuchus
Programmer
Posts: 1849
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Potential replacement of FOCS with Python

#52 Post by Ophiuchus »

o01eg wrote: Sat Oct 31, 2020 9:02 pm
Ophiuchus wrote: Sat Oct 31, 2020 7:57 pm I am not sure why you want to use AST though (if that is what you want). To save the bytecode conversion/time spent on interpretation?
Mostly because it could be unsafe to execute python code from untrusted sources and planned modding support could provide amount of untrusted sources.
legitimate!

I am a bit wary of using the AST instead of processing python structures - i think it would lead to lots of boilerplate and hard to maintain code (happy to be proven wrong)

we could use AST to restrict the program to a safe subset of python (e.g. no imports,file access,...) or even a minimal subset. bail out if an unsupported feature is encountered and interpret if the given program is safe.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12818
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Potential replacement of FOCS with Python

#53 Post by Geoff the Medio »

o01eg wrote: Sat Oct 31, 2020 9:02 pm...it could be unsafe to execute python code from untrusted sources and planned modding support could provide amount of untrusted sources.
Is it safe to parse code from untrusted sources?

ThinkSome
Dyson Forest
Posts: 245
Joined: Sun Mar 29, 2020 11:13 pm

Re: Potential replacement of FOCS with Python

#54 Post by ThinkSome »

If you are so concerned about unsafe code, then why not use lua, which is much simpler to execute in a protected way?

You could also require that all code used in multiplayer games is downloaded from a central server that you manage and not from the server, that way you can make sure both that the code is alright and that the code is available.

o01eg
Programmer
Posts: 1007
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#55 Post by o01eg »

Geoff the Medio wrote: Sat Oct 31, 2020 10:30 pm
o01eg wrote: Sat Oct 31, 2020 9:02 pm...it could be unsafe to execute python code from untrusted sources and planned modding support could provide amount of untrusted sources.
Is it safe to parse code from untrusted sources?
Except CVE in Python parser it's impossible to make changes in filesystem or send files with passwords via network with parsing.
Gentoo Linux x64, gcc-9.3, boost-1.74.0
Ubuntu Server 18.04 x64, gcc-7.4, boost-1.65.1
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 0.4.10.1.
Donations are welcome: BTC:14XLekD9ifwqLtZX4iteepvbLQNYVG87zK

o01eg
Programmer
Posts: 1007
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#56 Post by o01eg »

ThinkSome wrote: Sat Oct 31, 2020 11:39 pm If you are so concerned about unsafe code, then why not use lua, which is much simpler to execute in a protected way?

You could also require that all code used in multiplayer games is downloaded from a central server that you manage and not from the server, that way you can make sure both that the code is alright and that the code is available.
In case of adding new dependency we can use YAML
Gentoo Linux x64, gcc-9.3, boost-1.74.0
Ubuntu Server 18.04 x64, gcc-7.4, boost-1.65.1
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 0.4.10.1.
Donations are welcome: BTC:14XLekD9ifwqLtZX4iteepvbLQNYVG87zK

o01eg
Programmer
Posts: 1007
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#57 Post by o01eg »

There nice thread I've just read about it https://stackoverflow.com/questions/306 ... ure-python and it looks like there bunch of holes in tries to execute code.
Gentoo Linux x64, gcc-9.3, boost-1.74.0
Ubuntu Server 18.04 x64, gcc-7.4, boost-1.65.1
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 0.4.10.1.
Donations are welcome: BTC:14XLekD9ifwqLtZX4iteepvbLQNYVG87zK

ThinkSome
Dyson Forest
Posts: 245
Joined: Sun Mar 29, 2020 11:13 pm

Re: Potential replacement of FOCS with Python

#58 Post by ThinkSome »

o01eg wrote: Sun Nov 01, 2020 6:41 am
ThinkSome wrote: Sat Oct 31, 2020 11:39 pm If you are so concerned about unsafe code, then why not use lua, which is much simpler to execute in a protected way?

You could also require that all code used in multiplayer games is downloaded from a central server that you manage and not from the server, that way you can make sure both that the code is alright and that the code is available.
In case of adding new dependency we can use YAML
YAML is not a programming language, hence very limiting in what one can do.

User avatar
Vezzra
Release Manager, Design
Posts: 5467
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Potential replacement of FOCS with Python

#59 Post by Vezzra »

o01eg wrote: Sat Oct 31, 2020 9:02 pmMostly because it could be unsafe to execute python code from untrusted sources and planned modding support could provide amount of untrusted sources.
I'm confused - we already do execute Python code (from potentially untrusted sources) - the universe generation scripts, the turn events scripts and the AI scripts are all executed Python code.

And all that code resides in the "default" folder, meaning it belongs to the user customizable/moddable stuff, where also all the FOCS stuff is. Replacing FOCS with Python just migrates the part of the user costomizable stuff that is now scripted in FOCS to Python.

And the idea to replace FOCS wih Python is not to change the FOCS stuff to Python syntax and then again use some tools to parse that stuff ourselves, but to actually execute those Python scripts and construct all the condition and effects objects via Python (as we already use Python that way for the other stuff, so we're not really adding anything or any new dependencies, just reuse it to also do the things we now use FOCS for).

o01eg
Programmer
Posts: 1007
Joined: Sat Dec 10, 2011 5:46 am

Re: Potential replacement of FOCS with Python

#60 Post by o01eg »

Vezzra wrote: Sun Nov 01, 2020 12:21 pm I'm confused - we already do execute Python code (from potentially untrusted sources) - the universe generation scripts, the turn events scripts and the AI scripts are all executed Python code.

And all that code resides in the "default" folder, meaning it belongs to the user customizable/moddable stuff, where also all the FOCS stuff is. Replacing FOCS with Python just migrates the part of the user costomizable stuff that is now scripted in FOCS to Python.
That is why I proposed long ago to split it https://freeorion.org/forum/viewtopic.php?f=9&t=11538

Also if user doesn't host the game he doesn't execute all those python code and doesn't affected by its vulnerability.
Vezzra wrote: Sun Nov 01, 2020 12:21 pm And the idea to replace FOCS wih Python is not to change the FOCS stuff to Python syntax and then again use some tools to parse that stuff ourselves, but to actually execute those Python scripts and construct all the condition and effects objects via Python (as we already use Python that way for the other stuff, so we're not really adding anything or any new dependencies, just reuse it to also do the things we now use FOCS for).
As I understand if we have some

Code: Select all

def func():
    return 2+3
we cannot get freeorion's value ref

Code: Select all

Operation(OpType::PLUS, std::make_unique<Constant>(2), std::make_unique<Constant>(3))
directly from within python execution. We should either import script and call func in Python's interpreter or walk Python AST to convert it into FOCS value refs tree to be executed.
Gentoo Linux x64, gcc-9.3, boost-1.74.0
Ubuntu Server 18.04 x64, gcc-7.4, boost-1.65.1
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 0.4.10.1.
Donations are welcome: BTC:14XLekD9ifwqLtZX4iteepvbLQNYVG87zK

Post Reply