Potential replacement of FOCS with Python

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Post Reply
Message
Author
User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Potential replacement of FOCS with Python

#1 Post by adrian_broher »

EDIT by Vezzra: split from here.

Geoff asked me for some additional feedback regarding a potential replacement of FOCS with python.

First of all I would like to clarify what I consider 'FOCS'. 'FOCS' is the self-designed set of domain specific languages that is used to describe the game entities like ship designs, species, technologies, … and so on. The technical parts of it are
  • the text files, which contain the description of game entities in FOCS syntax. Those are located in the most recent development version in the `default/scripting` directory.
  • the internal `freeorionparser` library, which, as the name implies, contains the C++ implementation for the FOCS parser using the boost::spirit v2(? not quite sure about the version) library.
  • To some degree the C++ implementation of ValueReferences, Conditions and Effects as some FOCS syntax specifics 'bleed' into the design of those (e.g. the dot access syntax for dependent properties like 'Source.Owner', 'Target.SizeAsDouble', 'LocalCandidate.MaxStructure').
When I'm talking about 'removing FOCS' I mean:
  • Replace the text files with some other, to be determined, scripting language (python would be the obvious choice here, because it already exists).
  • Remove the 'freeorionparser' library.
  • Implement a some binding code to connect the ValueRef, Conditions and Effects with the new scripting language (with python that would be the boost::python binding library).
  • Remove existing FOCS bleeding in the ValueRef, Conditions and Effects code design.
While refactoring the FreeOrion code I came to the conclusion that there is no additional value to maintaining a domain specific language when we could instead use the other language we already use. Most of the critique isn't about the language quality but its existence and the integration into the other code. The problems I list are no particular order beside their personal annoyance factor.

The freeorionparser takes a lot of time to compile in the best case. When taking the current offers of build services like travisci or appveyor, which provide only a limited amount of build time for open source project, the general idea is not realizable. In the worst case we've already seen that the current code pushes the compiler implementation beyond it's limitations. On windows this manifests as compiler crash/stack overflow. So far we worked around this by splitting up the various compiler statements into multiple files which drives up the compile further. To throw in some numbers about the compile time (single build process without any cache mechanism used on my machine, using time make <targetname> in a build order that doesn't add up dependent targets, leaving out GiGi because it doesn't matter that much):
  • freeorioncommon: 10m 11.815s
  • freeorionparse: 24m 43.129s
  • freeoriond: 4m 14.960s
  • freeorionca: 3m 14.470s
  • freeorion: 18m 8.131s
Another problem is the fact that the 'common' domain model and the FOCS parser libraries have a circular linker dependency. The original plan was to build the freeorioncommon and freeorionparse libraries as shared object (.so,.dll,.dylib) to reduce memory usage, load time and installer size. Breaking the circular dependency on Linux and MacOSX this isn't such a big of a problem, because the symbols are not resolved during the linktime but when the dynamic linker resolves the symbols at runtime or the linker can be forced not to check the existence of all required symbols by setting a linker flag. On windows this can't be achieved easily without adding various build steps to create a so called export file, so after tinkering several weeks with that I gave up because it's not worth the effort in my opinion.

The final problem with using FOCS is the absence of any support tools. There is no IDE, no automatic documentation extraction, the debugging capabilities are crude. A lot of things need to worked around like getting the correct line number in case of a syntax error (done by putting the includes at the bottom of the file). That are things I expect from every language.

So what we currently have with FOCS:
  • An unmaintained Notepad++ syntax highlighting tool, which is probably way behind the current capabilities of FOCS.
  • A manually maintained documentation in the Wiki I don't know how outdated it is (dbenage-cx did a decent job according to the wiki page history).
  • No testing except the programmer smartness and luck. The automatic tests weren't touched for ages and they test only things like 'is this valid syntax' and not 'when writing this statement does it create the proper domain object' so I consider them nearly useless (we want to test our code, not the boost::spirit code).
When using an established language we don't need to maintain any support tools but can use what is already available (Andrey has already shown that extracting the python interface documentation is straight forward. The only thing missing is proper integration with the wiki or the existing Doxygen documentation).

About the 'FOCS is for non coders' argument and its variants:

I just can't agree with it. Consider the following use cases:

User A has never programmed a single line of code and is not able to abstract concepts like 'Target', 'EnqeueLocation', … even when looking at the manual. He will fail regardless of the language used.
User B has never programmed a single line of code but can abstract the game concepts with the help of the manual. He will struggle on minor things like using wrong attribute order, leaving out a quote and so on but he will get his change somehow into the game. The language doesn't matter as much as you would think because the user starts with zero knowledge and stumbling blocks are there in any language.
User C is proficient in general programming but does not know python. He has to learn either FOCS or python, depending on the scenario, to contribute to the project or change the game to his wishes.
User D is proficient in python will have no problem changing existing python code, but will need to learn FOCS, depending on the scenario.


About the syntax: the syntax is as complex or easy as we chose them to be (@MatGB don't consider existing python in the game as good reference to evaluate the complexity of the language in general. It tries to solve several complex problems and it's good but not perfect code. Also keep in mind that python gained a lot of it's popularity by being the 'first to learn'-language in schools and universities). For example:

Code: Select all

Special                                                                                    
    name = "WORLDTREE_SPECIAL"                                                             
    description = "WORLDTREE_SPECIAL_DESC"                                                 
    stealth = 0                                                                            
    spawnrate = 1.0                                                                        
    spawnlimit = 1                                                                         
    location = And [                                                                       
        Planet                                                                             
        Not Planet type = [Asteroids GasGiant Inferno]                                     
        Not Planet size = [tiny huge]                                                      
        Not WithinStarlaneJumps jumps = 2 condition = And [                                
            System                                                                         
            Contains And [                                                                 
                Planet                                                                     
                OwnedBy affiliation = AnyEmpire                                            
            ]                                                                              
        ]                                                                                  
    ]                                                                                      
    effectsgroups = [                                                                      
        EffectsGroup                                                                       
            scope = Source                                                                 
            activation = Source                                                            
            accountinglabel = "WORLDTREE_LABEL"                                            
            priority = [[LATE_PRIORITY]]                                                   
            effects = [                                                                    
                SetDetection value = Value + 10                                            
                SetMaxSupply value = Value + 1                                             
                SetTargetHappiness value = Value + 5                                       
            ]                                                                              
        EffectsGroup                                                                       
            scope = Source                                                                 
            activation = TargetPopulation low = 1                                          
            accountinglabel = "WORLDTREE_LABEL"                                            
            priority = [[LATE_PRIORITY]]                                                   
            effects = SetTargetPopulation value = Value + 1                                
        EffectsGroup                                                                       
            scope = And [                                                                  
                Planet                                                                     
                Not Source                                                                 
                OwnedBy affiliation = TheEmpire empire = Source.Owner                      
                Population low = 0.0001                                                    
            ]                                                                              
            effects = SetTargetHappiness value = Value + 1                                 
    ]                                                                                      
    graphic = "icons/specials_huge/worldtree.png" 
Could look in python like this (of course the c++ to python binding code must be written accordingly):

Code: Select all

class WorldTree(Special):
    name = "WORLDTREE_SPECIAL"
    description = "WORLDTREE_SPECIAL_DESC"
    stealth = 0
    spawnrate = 1.0
    spawnlimit = 1
    location = And(
        IsPlanet,
        Not(IsPlanetType([PlanetType.Asteroids, PlanetType.GasGiant, PlanetType.Inferno])),
        Not(IsPlanetSize([PlanetSize.TINY, PlanetSize.HUGE])),
        Not(WithinStarlaneJumps(jumps = 2, condition = And(
            System,
            Contains(And(
                IsPlanet,
                OwnedBy(Affiliation.AnyEmpire)
            ))
        )))
    )
    effectsgroups = [
        EffectsGroup(
            scope = Source,
            activation = Source,
            accountinglabel = "WORLDTREE_LABEL"
            priority = common.late_priority,
            effects = [
                Set("Detection", Add(Value, 10)),
                Set("MaxSupply", Add(Value, 1)),
                Set("TargetHappiness", Add(Value, 5))
            ]
        ),
        EffectsGroup(
            scope = Source,
            activation = InLimit("TargetPopulation", lower = 1),
            accountinglabel = "WORLDTREE_LABEL",
            priority = common.late_priority,
            effects = Set("TargetPopulation", Add(Value, 1))
        ),
        EffectsGroup(
            scope = And(
                IsPlanet,
                Not(Source),
                OwnedBy(Affilitation.TheEmpire, empire = Source.Owner),
                InLimit("Population", lower = 0.0001)
            ),
            effects = Set("TargetHappiness", Add(Value, 1))
        )
    ]
    graphic = "icons/specials_huge/worldtree.png"
Don't pin me on the exact syntax because this example was created offhand while keeping the existing FOCS syntax and the late evaluation of the conditions in mind. It certainly can be improved but should give you a general idea of what I had in mind.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13586
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Config files, FOCS, Python & Co

#2 Post by Geoff the Medio »

If that's roughly what the python-equivalent script would look like, I don't have any objections to the switch; it's pretty much the same script except for some punctuation.

I don't see how the python-equivalent is any easier to learn even for a python-expert (ignoring any potential IDE help) but it's also no worse.

Assuming that such a script could be written in python, how is it "run" what would happen when it's "run"? It seems to define a python class. How does that class get turned into the equivalent of what the boost::spirit parser produces now, or if it stays as a python class, what happens instead in order to run effects or populate the pedia with the scripted content?

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Config files, FOCS, Python & Co

#3 Post by adrian_broher »

Geoff the Medio wrote:Assuming that such a script could be written in python, how is it "run" what would happen when it's "run"? It seems to define a python class. How does that class get turned into the equivalent of what the boost::spirit parser produces now, or if it stays as a python class, what happens instead in order to run effects or populate the pedia with the scripted content?
The idea is to get the python type object for the specific class via reflection (asking for all subclasses of the base type 'Special', 'ShipPart', 'Species') and assigns the properties of this python type object to create the corresponding c++ object instance (Special containing the "WorldTree" description). For conditions and effects it would create instances of the corresponding C++ implementations ValueRef, Conditions, and Effects. Python only takes care of parsing and declaration, but is not involved in the evaluation of the statements. This is the reason why I replaced the Value + 4 with Add(Value, 4) and didn't keep it that was. This allows to create an C++ object Add instance, which can be evaluated at a later time with the initial values.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13586
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Config files, FOCS, Python & Co

#4 Post by Geoff the Medio »

adrian_broher wrote:The idea is to get the python type object for the specific class via reflection (asking for all subclasses of the base type 'Special', 'ShipPart', 'Species') and assigns the properties of this python type object to create the corresponding c++ object instance
Are you familiar with how to make such a request from C++ into python (using boost presumably)?

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Config files, FOCS, Python & Co

#5 Post by adrian_broher »

Geoff the Medio wrote:Are you familiar with how to make such a request from C++ into python (using boost presumably)?
Not yet. I plan to write a test application to check how easy the idea is to implement.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

xlightwavex
Space Kraken
Posts: 111
Joined: Mon Nov 16, 2015 5:57 am

Re: Potential replacement of FOCS with Python

#6 Post by xlightwavex »

I dunno sounds like a lot of work for little reward.

It also requires the average scripter to download a python editor or ide install it overcome problems with it and learn that editor as well.

Why not simply write a editor that automates this code even further and allows a type condition trigger action semantic all easily used from a editor then simply automate some of the code writing. The real novice doesn't have to do much at all then. While both tasks are work a editor pays off for the future regardless of the language used or the file types.
I was actually thinking of doing this just now or at least a little bit cause im getting annoyed doing copy paste work. I was just about to load all the script stuff from a copyied folder and some images or use text as a placeholder using c# and just condense the work done to go faster.

Or

Why not just work on the parser to make it automate many complex commands into simpler ones via adding new commands. I mean just look at the first few lines and translate it to normal human speach, and pick a command that does it.

here is a snippet of the original example.
class WorldTree(Special):
name = "WORLDTREE_SPECIAL"
description = "WORLDTREE_SPECIAL_DESC"
stealth = 0
spawnrate = 1.0
spawnlimit = 1
location = And(
IsPlanet,
Not(IsPlanetType([PlanetType.Asteroids, PlanetType.GasGiant, PlanetType.Inferno])),
Not(IsPlanetSize([PlanetSize.TINY, PlanetSize.HUGE])),
Not(WithinStarlaneJumps(jumps = 2, condition = And(
System,
Contains(And(
IsPlanet,
OwnedBy(Affiliation.AnyEmpire)
))
)))
)
The below part is cool its simple enough
class WorldTree(Special):
name = "WORLDTREE_SPECIAL"
description = "WORLDTREE_SPECIAL_DESC"
stealth = 0
spawnrate = 1.0
spawnlimit = 1
This is the only part that needs simplified.
location = And(
IsPlanet,
Not(IsPlanetType([PlanetType.Asteroids, PlanetType.GasGiant, PlanetType.Inferno])),
Not(IsPlanetSize([PlanetSize.TINY, PlanetSize.HUGE])),
Not(WithinStarlaneJumps(jumps = 2, condition = And(
System,
Contains(And(
IsPlanet,
OwnedBy(Affiliation.AnyEmpire)
))
)))
)
Whats the above say if i were to guess.

I would do so by showing what this would look like in the starcraft 1 editor system which was very popular. Id say it would look like this.

Trigger... On_Map_Creation, Conditions... {
If_Type... current_planet, is_not... PlanetType.Asteroids, PlanetType.GasGiant, PlanetType.Inferno.
If_Type... current_planet, is_not... PlanetSize.TINY, PlanetSize.HUGE.
If_Type... current_planet, is_not... Within_Distance_Of...(Value... 2, Of_Type... planet, OwnedBy... AnyOne).
If_Random_Less_Then...(5, 100)
},
Action... Create, Amount... 1, Category... planet_special, Name... 'world_tree'.
Action... Done.

Here in the editor you had a choice to select commands shown by italicized words you had a list to select from as available commands and then you entered a value, the value was also selectable. This continued on and only gave you available commands as you moved thru each step of the triggers conditions and actions. The editor itself built the parse-able code from that. For example the condition bracket itself {... was a selection that could instead be none.

It's something to think about as anyone could write a editor regardless of the language. As well even a novice can figure things out from a set of drop downs as he continues.

This makes me think of the 'Translated Thought' tech. The human sees the object or thing he speaks of first then logically processes the values associated to it and the conditions of those values. The computer code the truth or falseness of the proceeding statements.
Higher level languages are always easier to use for the user and as long as its not compiled during runtime its just fine for the computer too.
Last edited by xlightwavex on Fri Sep 09, 2016 9:20 pm, edited 6 times in total.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Potential replacement of FOCS with Python

#7 Post by adrian_broher »

I dunno sounds like a lot of work for little reward.
Well, the invested work is mainly to fix some (in my eyes) serious design flaws in the program architecture. The parser makes the build unnecessary hard to maintain, for more details check out the opening post.

But TL;DR: I want to get rid of the FOCS compiler and replace it with ${something else}, most preferable python because we already use it.
It's something to think about as anyone could write a editor regardless of the language.
The idea of an editor (graphical or not, with some optimizing compiler or not) ist surely interesting, but orthogonal to the problem I face and want to solve.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

xlightwavex
Space Kraken
Posts: 111
Joined: Mon Nov 16, 2015 5:57 am

Re: Potential replacement of FOCS with Python

#8 Post by xlightwavex »

adrian_broher wrote: Well, the invested work is mainly to fix some (in my eyes) serious design flaws in the program architecture. The parser makes the build unnecessary hard to maintain, for more details check out the opening post.

But TL;DR: I want to get rid of the FOCS compiler and replace it with ${something else}, most preferable python because we already use it.

The idea of an editor (graphical or not, with some optimizing compiler or not) ist surely interesting, but orthogonal to the problem I face and want to solve.
I've been dabbling with the script files playing around, upon realizing how difficult it can be just to alter the tech tree. The last couple days i just started writing out some code to parse the tech files into a app for a basic demo editor. Not sure if i even want a editor of just a way to chart analyze the tech tree via time cost and jumps anyways.
Biggest problem so far is the effects group syntax. I would have no clue on how to actually parse that portion out into running code atm. So i just copyed the whole portion of it in place. The rest i have pulled out into variables.
Looking thru the py files though, it looks like part of the tech tree is in place for the Ai python code. How tightly coupled is that to the tech tree ? Are the python files parsed or compiled ?

If you change up the scripting it might be a good thing. Can you give out a bit more info on what this will change as well e.g. ...

Will the directory structures remain the same ?
What sort of changes will be made to files in directory's, will some become no longer necessary ect ?
Do you think it will make it possible to simplify the scripting syntax itself so that its clearer / easier to use ?
Will it be possible to get better error output ? Every time there is a error the console window disappears.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Potential replacement of FOCS with Python

#9 Post by adrian_broher »

xlightwavex wrote:I've been dabbling with the script files playing around, upon realizing how difficult it can be just to alter the tech tree. The last couple days i just started writing out some code to parse the tech files into a app for a basic demo editor. Not sure if i even want a editor of just a way to chart analyze the tech tree via time cost and jumps anyways.
So your use case is to find the most effective tech development? What do you mean with 'jumps'? The total cost and time for a technology with its prequisites?
xlightwavex wrote:Biggest problem so far is the effects group syntax. I would have no clue on how to actually parse that portion out into running code atm. So i just copyed the whole portion of it in place. The rest i have pulled out into variables.
Well, it's not straight forward way to parse it. We use the boost::spirit v2 C++ library to do this. If you're not willing to reuse that or to write/use a LL(∞) parser reimplementing the rules defined in the parse/ implementation you're out of luck.
xlightwavex wrote:Looking thru the py files though, it looks like part of the tech tree is in place for the Ai python code. How tightly coupled is that to the tech tree ?
As far as I know the AI only interacts with the tech tree via the domain model, it doesn't parse any of the FOCS files. Maybe an AI programmer can give an better answer regarding the interaction between tech tree and AI.
xlightwavex wrote:Are the python files parsed or compiled ?
I don't get how this question is related to the rest of your posting. Parsing is done for every programming language, it's the translation from a 'lump of text' into a some structured data; for programming languages it's usually an abstract syntax tree. Compiling is an ambiguous term. Usually people mean by compiling 'translating source code into machine instructions' but any program that translates a 'higher level' into a more simple one is a compiler (e.g. C to assembler, assembler to machine instructions, java to JVM bytecode, C# to Common Intermediate Language, Python to Python bytecode). You probably want to know if the Python is interpreted or compiled to machine instructions? As far as I know python compiles the scripts to an intermediate bytecode and interprets this bytecode in an virtual machine (like a 'python CPU' with a 'python machine instruction set') so it's interpreted in my definition.
xlightwavex wrote:Will the directory structures remain the same ?
What sort of changes will be made to files in directory's, will some become no longer necessary ect ?
Do you think it will make it possible to simplify the scripting syntax itself so that its clearer / easier to use ?
Will it be possible to get better error output ? Every time there is a error the console window disappears.
I can't answer that yet, because it's still the 'collect ideas and tinker' phase and some of your questions are not related with the parser itself. So far only two goals are defined.
  • Replace the FOCS scripts with python scripts.
  • Keep the syntax of python as close as possible to FOCS also keep it declarative as possible.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

Morlic
AI Contributor
Posts: 296
Joined: Tue Feb 17, 2015 11:54 am

Re: Potential replacement of FOCS with Python

#10 Post by Morlic »

adrian_broher wrote:
xlightwavex wrote:Looking thru the py files though, it looks like part of the tech tree is in place for the Ai python code. How tightly coupled is that to the tech tree ?
As far as I know the AI only interacts with the tech tree via the domain model, it doesn't parse any of the FOCS files. Maybe an AI programmer can give an better answer regarding the interaction between tech tree and AI.
The AI parses no content (FOCS) files directly. Any information it gets is either queried from the server using the C++ AIInterface or is hardcoded in the .py files.

For instance, currently the tech paths of the AI is hardcoded: It basically has a list of tech names it tries to research in a specified order unless certain conditions are met and other (hardcoded) techs are fast-tracked. If you change the name of a tech (or add another one), the AI will not research it unless it is a prerequisite for a later tech it has in its foreseen tech tree (or has researched all of those already).
If you edit the effect of techs, the AI research path might make no sense anymore as it currently does not know the effects of a tech (unless specified in the .py files, i.e. hardcoded).

There also are some decisions being gated by having certain techs researched and hardcoded effects of techs in the code (for example population boni from techs) mapped by the tech name.

As far as I know, the AI shouldn't crash or throw errors (except for initial consistency checks/warnings) if techs are modified (if it does, please report so it can be fixed). Its performance, however, may be critically impaired depending on the changes unless you update the AI scripts as well. That shouldn't stop you from doing it - adjusting the AI to a playable level should be quite easy unless you introduce some really gamechanging stuff that requires complex decision-making (if you need help with that, feel free to ask).
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

xlightwavex
Space Kraken
Posts: 111
Joined: Mon Nov 16, 2015 5:57 am

Re: Potential replacement of FOCS with Python

#11 Post by xlightwavex »

Morlic wrote:
adrian_broher wrote:
xlightwavex wrote:Looking thru the py files though, it looks like part of the tech tree is in place for the Ai python code. How tightly coupled is that to the tech tree ?
As far as I know the AI only interacts with the tech tree via the domain model, it doesn't parse any of the FOCS files. Maybe an AI programmer can give an better answer regarding the interaction between tech tree and AI.
The AI parses no content (FOCS) files directly. Any information it gets is either queried from the server using the C++ AIInterface or is hardcoded in the .py files.

For instance, currently the tech paths of the AI is hardcoded: It basically has a list of tech names it tries to research in a specified order unless certain conditions are met and other (hardcoded) techs are fast-tracked. If you change the name of a tech (or add another one), the AI will not research it unless it is a prerequisite for a later tech it has in its foreseen tech tree (or has researched all of those already).
If you edit the effect of techs, the AI research path might make no sense anymore as it currently does not know the effects of a tech (unless specified in the .py files, i.e. hardcoded).

There also are some decisions being gated by having certain techs researched and hardcoded effects of techs in the code (for example population boni from techs) mapped by the tech name.

As far as I know, the AI shouldn't crash or throw errors (except for initial consistency checks/warnings) if techs are modified (if it does, please report so it can be fixed). Its performance, however, may be critically impaired depending on the changes unless you update the AI scripts as well. That shouldn't stop you from doing it - adjusting the AI to a playable level should be quite easy unless you introduce some really game changing stuff that requires complex decision-making (if you need help with that, feel free to ask).
MICRO_MANUF.focs // <<<<<<<<<<

Code: Select all

Tech
    name = "PRO_MICROGRAV_MAN"
    description = "PRO_MICROGRAV_MAN_DESC"
    short_description = "INDUSTRY_SHORT_DESC"
    category = "PRODUCTION_CATEGORY"
    researchcost = 50 * [[TECH_COST_MULTIPLIER]]
    researchturns = 5
    prerequisites = [ 
	    "CON_ORBITAL_CON"
		"PRO_INDUSTRY_CENTER_I"
		]
    effectsgroups = [
        EffectsGroup
            scope = And [
                ProductionCenter
                OwnedBy empire = Source.Owner
                ContainedBy And [
                    System 
                    Contains And [
                        Planet
                        Planet type = Asteroids
                        OwnedBy empire = Source.Owner
                    ]
                ]
                Focus type = "FOCUS_INDUSTRY"
            ]
            priority = [[VERY_LATE_PRIORITY]]
            effects = SetTargetIndustry value = Value + 5
    ]
    graphic = "icons/tech/microgravity_manufacturing.png"

#include "/scripting/common/base_prod.macros"
#include "/scripting/common/priorities.macros"
AIDependencies.py

Code: Select all

# Please see the Note at top of this file regarding PlanetSize-Dependent-Lookup
# building supply bonuses are keyed by planet size; key -1 stands for any planet size
building_supply = {
    "BLD_IMPERIAL_PALACE": {
        -1: 2,
    },
    "BLD_MEGALITH": {
        -1: 2,
    },
    "BLD_SPACE_ELEVATOR": {
        fo.planetSize.tiny: 1,
        fo.planetSize.small: 2,
        fo.planetSize.medium: 3,
        fo.planetSize.large: 4,
        fo.planetSize.huge: 5,
        fo.planetSize.gasGiant: 4,
    },
}

# tech names etc.
GRO_LIFE_CYCLE = "GRO_LIFECYCLE_MAN"
PRO_ORBITAL_GEN = "PRO_ORBITAL_GEN"
PRO_SOL_ORB_GEN = "PRO_SOL_ORB_GEN"
PRO_MICROGRAV_MAN = "PRO_MICROGRAV_MAN" // <<<<<<<<<<<<<<<<<<<<<
PRO_SINGULAR_GEN = "PRO_SINGULAR_GEN"
PROD_AUTO_NAME = "PRO_SENTIENT_AUTOMATION"
So we have a focs with scripting code that is akin to a programming algorithm (instead of akin to method calling passed by values) e.g. EffectsGroups... Then we have py's with sets of hardcode data structures that look like they belong in a focs that is editable. Those will botch up the ai if someone scripts techs unknown to them.

Im wondering if part of the problem here is lack of clarity for what data should be in which files as well as what files should be responsible for what.

Just shooting off the hip but im thinking the original purpose of having both the focs files and the py's. Was to clarify what is editable or should be by anyone. Opposed to what should not be, is that correct ?
So what we currently have with FOCS:

An unmaintained Notepad++ syntax highlighting tool, which is probably way behind the current capabilities of FOCS.
A manually maintained documentation in the Wiki I don't know how outdated it is (dbenage-cx did a decent job according to the wiki page history).
No testing except the programmer smartness and luck. The automatic tests weren't touched for ages and they test only things like 'is this valid syntax' and not 'when writing this statement does it create the proper domain object' so I consider them nearly useless (we want to test our code, not the boost::spirit code).
F-AQ

How will this change help to get the above corrected ?
What do we get after ?
What do we lose ?

Im trying to figure out the end result of such a change if or how it changes the game for the scriptor, as i believe the focs files are aim directly towards novice scripters, which includes me for both focs and py lol.

Morlic
AI Contributor
Posts: 296
Joined: Tue Feb 17, 2015 11:54 am

Re: Potential replacement of FOCS with Python

#12 Post by Morlic »

xlightwavex wrote: So we have a focs with scripting code that is akin to a programming algorithm (instead of akin to method calling passed by values) e.g. EffectsGroups... Then we have py's with sets of hardcode data structures that look like they belong in a focs that is editable. Those will botch up the ai if someone scripts techs unknown to them.

Im wondering if part of the problem here is lack of clarity for what data should be in which files as well as what files should be responsible for what.
From my perspective it is very clear what should be in the AI script files - everything that the AI needs to know that it can not query dynamically from the server. That is not a question of design but a question of practicability.

FOCS allows very complex effects and conditions. It is (from my perspective) unrealistic to teach the AI the entire ruleset that FOCS allows. Understanding every single effect and condition is not a realistic goal for the AI.

What we can aim for is a selected, basic rule set that the AI understands and based on which it can make basic decisions - thus minimizing the amount of hardcoded values. One example are species traits: Each species may have some research and industry modifier that can take discrete number of values. The AI understands that easily.
Another example is the ShipDesignAI I wrote that lets the AI dynamically assess the strength of ship parts and find "optimum" designs based upon a mathematical well defined target function. That works, and only works, because there is a simple ruleset that is implemented: Weapons give x damage. Armour parts give y structure. Shields give, well, z shieldstrength and do not stack.

Any deviation from that ruleset (e.g. a part has a secondary effect) must still be somehow taught to the AI. Now you could envision to write complex parsing that perfectly understands the condition and effect which may take hours/days/years depending on the generality of your approach.
Or you could write a 2-line-statement in possibly as low as 1 minute:

Code: Select all

if partname == AIDependencies.SPECIAL_PART_NAME:
    self.stealth += 20
Does it rely on hardcoded, potentially changing data? Yes. Is there a clear, maintainable, easy-to-use alternative? None that I see. But feel free to suggest some.

Of course, you could replace the querying for name by adding a certain tag to a tech in FOCS and then looking for it. That tag would be single-use, rely strictly on the underlying AI implementation and would absolutely be not understandable/meaningful for any content scripter. Unless he knows the underlying AI code. In which case he could just as well edit the AI scripts. Doesn't really seem like an improvement to me and specifically does not address any of the problems that arise from deleting/adding/changing a tech whatsoever.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

xlightwavex
Space Kraken
Posts: 111
Joined: Mon Nov 16, 2015 5:57 am

Re: Potential replacement of FOCS with Python

#13 Post by xlightwavex »

Code: Select all

if partname == AIDependencies.SPECIAL_PART_NAME:
    self.stealth += 20
Well in reply im just making a point and the point is the AI just knows to build it early or late because some .py file told it some were its pretty gud. Get it about such and such time. That makes your ai extremely limited it makes scripting somewhat pointless unless your aware of it.
In that case it (the 'ai should get it cause its gud' file) should be in focs AI_BUILD_ORDER file or something like that. You could have one for each race or race types you could randomly attach to a race on map start. The point is... 'what is were' ... in the focs or py files, content wise.


Anyways this has me thinking about something more interesting...

A addition point to consider that all this below in the focs is actually, in reality....
effectsgroups = [
EffectsGroup
scope = And [
ProductionCenter
OwnedBy empire = Source.Owner
ContainedBy And [
System
Contains And [
Planet
Planet type = Asteroids
OwnedBy empire = Source.Owner
]
]
Focus type = "FOCUS_INDUSTRY"
]
priority = [[VERY_LATE_PRIORITY]]
effects = SetTargetIndustry value = Value + 5
]
Just this...

'''Increases industry by +5 on all Industry-focused colonies in system with an asteroid belt outposts or colonies.

None that I see. But feel free to suggest some.
It's far easier to envision what you would love to simply type into the focs file. It's fun then to spend the effort on imagining how you can make it happen.

For a average user it should look like this i think.

Code: Select all

EffectsGroup
[
CurrentPlayer_IndustryFocusIncrease_AllOwnedWorlds_InSameSystemWith( 5, "Planet_Type_Asteriod" );
]
To emphasize.

EffectsGroup
[
CurrentPlayer_IndustryFocusIncrease_AllOwnedWorlds_InSameSystemWith( 5, "Planet_Type_Asteriod" );
]
  • Whats the result of this code wise, if we could imagine a way to code it, that is logical.
    Since the effect is implicit the below line is not required anymore.
    effects = SetTargetIndustry value = Value + 5.
    The whole line is both the keyword and instructions.
In this case the ai can simulate the effect because it knows how to implicitly determine the value of its worth by formula. The number of industry_worlds it has in a system with a Asteriod * 5; is the industrial value of the tech.

So how i would do that is to make a bunch of commands in a logical manner that anyone could look up and use quickly and in a intuitive way.

In this case the scripted keyword actually denotes a c++ function.
CurrentPlayer.IndustryFocusIncrease.AllOwnedWorlds.InSameSystemWith(...){...}
That is within a series of nested inner class that your actually calling, like so with a logical repeating structure. By calling the effect function.

Code: Select all

class Effects 
{
  class [u]Current[/u]Player{ 
     class IndustryFocusIncrease{ 
        class AllFocusedProdWorlds{ m(value) 
           class ExceptSameSystemWith{m( system_type_value ) }
           class InSameSystemWith{ m( system_type_value ) }
        }
        class AllOwnedWorlds{ m(value) 
           class ExceptSameSystemWith{ m( system_type_value ) }
           class InSameSystemWith{ m( system_type_value ) }
        }
        class AllOwnedProductionWorlds{ m(value) 
           class ExceptSameSystemWith{ m( system_type_value ) }
           class InSameSystemWith{ m( system_type_value ) }
        }
     }
  }
  }
  class [u]All[/u]Players{ 
     class IndustryFocusIncrease{ 
        class AllFocusedProdWorlds{ m(value) 
           class ExceptSameSystemWith{m( system_type_value ) }
           class InSameSystemWith{ m( system_type_value ) }
        }
        class AllOwnedWorlds{ m(value) 
           class ExceptSameSystemWith{ m( system_type_value ) }
           class InSameSystemWith{ m( system_type_value ) }
        }
        class AllOwnedProductionWorlds{ m(value) 
           class ExceptSameSystemWith{ m( system_type_value ) }
           class InSameSystemWith{ m( system_type_value ) }
        }
     }
  }
}
The entire set of classes is the effect by the time its done its basically queryed a logical set of searches and the effect is applied to the result or to a test result the ai can evaluate compared to other possible choices. A cost effectiveness comparison. The whole chain of classes forms a script-able keyword that can be used to call a built in effect. Even at bare minimum the effects value of when to get it could be hard coded separate from the tech itself decoupling it ideologically.

That's not to say we shouldn't have custom effects just that the most heavily used ones should be turned into basic effects that are simple common one shot calls. that the ai can compare and that can be used to quickly tact right onto a tech or other thing.

That we should have Basic Effects.

Im not familiar with all the jargon, though i got the sense macros do something similar already.

spikethehobbit
Space Squid
Posts: 66
Joined: Mon Aug 27, 2012 7:24 pm

Re: Potential replacement of FOCS with Python

#14 Post by spikethehobbit »

The current FOCS implementation is rather inefficient. One problem we will face is that the best format to write the data files in is not a form that the server can use directly. Python is actually a good choice here because it supports manipulating complex data structures.
All contributions are submitted under GPL or LGPL v2 or later, or under appropriate Creative Commons licence, consistent with project guidlines.

User avatar
em3
Vacuum Dragon
Posts: 630
Joined: Sun Sep 25, 2011 2:51 pm

Re: Potential replacement of FOCS with Python

#15 Post by em3 »

My 2c: Value could be made of type with __add__ and __radd__ methods overloaded, so that code like this would also be valid:

Code: Select all

effects = [
                Set("Detection", Value + 10),
                Set("MaxSupply", Value + 1),
                Set("TargetHappiness", Value + 5)
            ]
EDIT: I think that also defining __setattr__ for the type of Source could enable doing something like this:

Code: Select all

effects = [
                Source.Detection = Value + 10,
                Source.MaxSupply = Value + 1,
                Source.TargetHappiness = Value + 5
            ]
Of course, __setattr__ would have to return the same thing, that Set was supposed to return (to place in effects list).

EDIT2: Note, that addition and setting would not happen in python, it would just generate the appropriate effect structures.
https://github.com/mmoderau
[...] for Man has earned his right to hold this planet against all comers, by virtue of occasionally producing someone totally batshit insane. - Randall Munroe, title text to xkcd #556

Post Reply