Config files, FOCS, Python & Co

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Message
Author
User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Config files, FOCS, Python & Co

#1 Post by Vezzra »

I continue a discussion that started in a PR on github here, as it belongs more on the forum than there:
LGM-Doyle wrote:Speaking of "configuration files", there are at least 8 locations for/types of "configuration files": FOCS, config.xml, persistent_config.xml, universe_tables.py, AIDependencies.py, EnumsAI.py, buildings.inf and config.ini. They use at least 4 different syntaxes: FOCS, XML, python, ini and line terminated double quote delineated file of strings. Some are not visible to the server. And some duplicate or override information from other configuration files or the server.

Are things supposed to be moving in one direction or another?
Well, *sigh*. You're right of course, the current state of things in that department isn't exactly exhilarating. The result of many people working over the course of many years and throwing in temporary solutions when they wanted something to be configurable but didn't have the time to implement a proper config solution plus corresponding GUI.

It's bothering me too, and we definitely need to do something to consolidate this mess. That said, all these "config" files/mechanics address very different things, so even after a proper cleanup/consolidation we will have several different types of them.

FOCS serves the purpose of defining the game content. The "config.xml" is a storage for all kinds of parameters like GUI/appearance etc., so we most likely won't be able to unify these two.

But, for example, one thing I mulled over a lot in the past is if we really need both FOCS and Python. While it's obviously impossible to use FOCS to implement the AI (or even universe generation as it's done now), it would certainly be possible to do all the content definitions in Python. We could throw out the entire parser code (for all its power the Boosts parser lib is a template nightmare after all) and scripters would have one language less to worry about when customizing FO.

But then again, OTOH, there are probably good reasons to stick with both. FOCS is supposed to be easily readable, understandable and learnable even for non-coders. Now, I can't really tell if that's true, because for me Python is easier to read and understand than FOCS. But apparently most other people do better with FOCS, and it seems that it won't be easy to set content definition up in Python in a way that's as easily comprehensible to non-coders. And that's just the minor obstacle here - the main advantage FOCS has over Python is performance I guess. The FOCS scripts are read and parsed once, and transformed into data structures that can be processed very efficiently. If all the content was implemented in Python, that Python code would have to be executed (and this means interpreted) each time it's invoked, and I don't even want to imagine the hit on performance this means (*shudder* :shock:). (Geoff, correct me if I'm wrong.)

So, because of the very different purposes these languages serve, we probably have to keep both of them (unless someone comes up with some ingenious idea how to do all the FOCS stuff in Python without the performance hit).

If you have ideas how to simplify our current config file maze, you're certainly welcome to share them. :)

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Config files, FOCS, Python & Co

#2 Post by adrian_broher »

If you have ideas how to simplify our current config file maze, you're certainly welcome to share them. :)
Kill FOCS, use python. End of story.
FOCS is supposed to be easily readable, understandable and learnable even for non-coders.
I tried to understand it and I'm still failing. And I feel proficient in C, C++, Python, Java, Bash, SQL and other languages.
The FOCS scripts are read and parsed once, and transformed into data structures that can be processed very efficiently. If all the content was implemented in Python, that Python code would have to be executed (and this means interpreted) each time it's invoked, and I don't even want to imagine the hit on performance this means (*shudder* :shock:).
See #797 for example. The problem are not (premature)optimizations, but most likely naive algorithms used all through the code base.
Last edited by adrian_broher on Tue Aug 09, 2016 6:19 pm, edited 1 time in total.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
MatGB
Creative Contributor
Posts: 3310
Joined: Fri Jun 28, 2013 11:45 pm

Re: Config files, FOCS, Python & Co

#3 Post by MatGB »

I am a complete non-coder, I learnt BASIC as a kid and can bash together a bit of basic HTML.

I can read, adapt and code in FOCS, and was doing so way before I even joined the forum.

Python is a weird thing involving weird syntax that my brain still has trouble reading. I am sure I could learn it.

I didn't need to learn FOCS to start scripting. I can only speak for myself, but I'm 99% certain I wouldn't be that involved if at all with the project if it had meant learning to read Python in order to change the basic effects I started with messing with (IIRC, back when supply was infrastructure based I went in and added a bunch of boost modifers to that to make the game, well, playable)
Mat Bowles

Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

LGM-Doyle
Programmer
Posts: 219
Joined: Mon Feb 29, 2016 8:37 pm

Re: Config files, FOCS, Python & Co

#4 Post by LGM-Doyle »

This was not exactly the direction I was expecting.

I was concerned with three things:
- all configuration files/data be in standard location(s). Easy for people to find.
- all configuration be done by a single chunk of code. Avoids two different parses of the same data.
- all configuration be done in one executable and then be piped out to the clients. Avoids client and server being configured differently.

I broke all of those rules by adding a single line of configuration information in a random python
file, in a client executable. Vezzra called me on it and it is fixed.


However, on to the discussion at hand.

MatGB you are no longer a complete non-coder.

You have been publishing reviewed code for over a year. Congratulations.

I get your point. Python would have been too large a barrier for you to start contributing.



adrian_broher, thanks for complimenting PR #797.

I agree with what you are saying, a more experienced programmer sees FOCS and can tell that it is
non-standard with the problems that implies. Learning another non-standard scripting language just
to add content is a barrier to someone with experience.


Vezzra I may restate some of the ideas from your post.


First, FOCS is working as a content data and scripting language. So if nothing is done it keeps
working.


For the following I'm using these definitions for data and scripting in FOCS.

Data are simple mappings like 'name ="SOME_NAME"' or 'graphic="some_file.png"'. Everyone willing to
modify freeorion will be willing to change "some_file.png" to "some_other_file.png".

Lists are also simple enough to count as data.

Scripting is everything else: boolean statements, conditions, effects.


Part of the fun of freeorion is re-configuring stuff and seeing what happens. Adding new
species/graphics/story text is fun. If freeorion keeps the barrier low we get more contributed
content.

An artist looking to add a new image data wants a line that says `filename: <blah>`. Any
unfamiliar symbols "]}!?=>$%" may make them uncertain enough to not try.

FOCS is a large barrier for an author/artist with no programming experience. The boolean logic
clauses require the player to have some math or a programming background to understand. FOCS may
have started as a simple data format, but it has expanded to Turing completeness, as evinced by the
clever Experimentors implementation.


Moving all the data and scripting from FOCS to python makes the data less accessible to more of
the potential audience.

As Vezzra implied, even an "if/then/else" statement is a foreign language to most people.


Just moving the scripting portion of FOCS from FOCS into python would make data in FOCS more
accessible to non-programmers and make the scripting more accessible to programmers.

The just above hides all the complexity of moving the description of the scripting from FOCS
to python while keeping the execution in C++.

That would leave FOCS as simply a data format. Freeorion could choose a non-custom data-only format
that is accessible to non-programmers.

I suggest YAML. [1] [2].
It can support all the behavior of a data only FOCS and in config.xml. YAML is a simpler syntax.

Ideally all the configuration files are in standard formats and freeorion can stop maintaining
custom parsers.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Config files, FOCS, Python & Co

#5 Post by Geoff the Medio »

Vezzra wrote:...the main advantage FOCS has over Python is performance I guess. The FOCS scripts are read and parsed once, and transformed into data structures that can be processed very efficiently.
If set up appropriately, Python could be used to construct the same sort of data structures as are created from FOCS parsing.
If all the content was implemented in Python, that Python code would have to be executed (and this means interpreted) each time it's invoked, and I don't even want to imagine the hit on performance this means (*shudder* :shock:). (Geoff, correct me if I'm wrong.)
It's not clear what is meant by writing content scripts in Python. Currently all the conditions and effects and value references that are combined / composed to implement in-game content are written in C++, and are assembled into data structures by parsing. If they were instead assembled into data structures by Python code that is run once at startup, instead of parsing FOCS, it shouldn't affect the during-the-game performance at all. If they were all replaced by Python code that itself modifies the gamestate, that would potentially have major performance effects, and would be a huge re-implementation of existing engine functionality.

But regarding the "main advantage" noted above, I'd say the main advantage of FOCS over Python is that it's structures to reflect the way game content is implemented in the engine. You need costs and production times, names and descriptions, graphics, location or enqueue conditions, and various effectsgroups, which in turn need scope and activation conditions and effects, which all in turn need to reference the gamestate using valuerefs and nested conditions, etc. The somewhat rigid structure the FOCS parser imposes corresponds to that.

If there's a problem with readability of FOCS, it's likely due to the heavy use of macros and potentially complicated logic that some effects / conditions requires. The macros actually make many script files much shorter, and easier to modify for thing like graphics, eg.

Code: Select all

icon = "icons/ship_parts/colony_pod.png"
. But those same macros also hide away the meat of the game-logic that the content uses to work, making it substantially harder to modify (with the advantage of avoiding code duplication), particularly after increasing using of macros with parameters and nested macros. And yet, I don't think switching to Python would have any benefit in those regards, as you'd still have to write or modify a line specifying what file to use for some bit of content's graphic, or its cost, which will look nearly identical to the FOCS equivalent. Python would similarly end up using common data structures to compose C++-implemented conditions/effects/valuerefs, or would have to re-implement them all and then use python-style composition of functions to similarly avoid code duplication. And due to the complicated logic needed for some content, those python functions or condition/effect/valueref compositions will be just as complicated as they are in FOCS.

Ophiuchus
Programmer
Posts: 3433
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Config files, FOCS, Python & Co

#6 Post by Ophiuchus »

Just my 2 cents.
For me python is easy to understand, but having a scripting language with game-related terms is more important. Usually i am a big fan of embedding domain specific languages in a general purpose language, so the common ground can be easily understood by people with the right background.

In the freeorion case the structure of FOCS (and the underlying c structure i suppose) is more of a declarative/rule-based language (declaring an activation set of conditions, a set of targets' conditions and a set of effects to be processed) than python. You need to understand these concepts regardless of the used syntax/language.
The common ground could be the effects (imperative changes to state), but those are translated to the c structures which get interpreted(?) when running freeorion. (Not sure,... I think I need to read more of that code)

FOCS is IMHO not as easy as it could be and looks to be expanded on an ad-hoc basis, but it works for many cases.
As such for me a more pressing issue would be to speed up the development process. To evaluate my scripting changes it takes me on my laptop one to three minutes (restarting freeorion, loading and starting a game with the test setup, advancing a turn and check the results) to see what I did wrong.
In other environments I would script some automated tests which would tell me that in the matter of seconds.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Config files, FOCS, Python & Co

#7 Post by Geoff the Medio »

Ophiuchus wrote:FOCS is IMHO not as easy as it could be...
Can you be specific?
...and looks to be expanded on an ad-hoc basis
Pretty much.

Ophiuchus
Programmer
Posts: 3433
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Config files, FOCS, Python & Co

#8 Post by Ophiuchus »

Geoff the Medio wrote:
Ophiuchus wrote:FOCS is IMHO not as easy as it could be...
Can you be specific?
Being more specific is hard. Being vague, it is mostly the gut feeling that you get bitten too often and having many questions which the naming/syntax structure did not answer.

I'll try to be as specific as I can. I was now scripting specials and ship parts.

Questions which arose were:
1) what exactly is the Source (e.g. planet for the special, ship for the ship part)
2) how often does the code get checked/the effects get executed (e.g. if having a ship part n times on the ship - once per ship or n times per ship?)
3) stackinggroup -> What is a stackinggroup? is that like a tag attached to every object in the activation condition? is that attached to the set of objects?
4) only way to reuse definitions is via macros
4a) bundling of effects on different scope is not possible (e.g. i wanted in certain circumstances: a) unload the supply ship (effect to planet industry) and b) destroy the supply ship (effect to ship) )
5) no way to reuse the set of items in scope in the effects; e.g. for applying effects to a subset
6) not everything is an object (so, e.g. the interface for handling specials is different),

I think naming stroke me often as not helpful:
1) 'location' -> maybe buildlocation; or putting buildcost, buildtime, and location in a own subsection 'build'
2) there were the same things called differently in different contexts.. (cant remember ATM). Also that scope would be better targets.

Maybe I can convey some ideas by code. I find the following syntax more helpful than the current:

Code: Select all

Part
    name = "CO_SUPPLIES_030"
    description = "CO_SUPPLIES_DESC" 
    class = General
    capacity = 30
    mountableSlotTypes = [External Internal]
    build [ 
        buildCost = 70
        buildTime = 5
        onlyBuildOnPlanet = OwnedBy empire = Source.Owner
    ]
    forEachPartOnTheShip [ // EffectsGroups named accordingly to item type (e.g. forEachPlanetWithThisSpecial), indicating how often this gets executed
        ifSourceShip [ inSystem ] then [ // EffectsGroup/activation condition named accordingly to Source type. 
            ifTargets [ Planet ; InSystem id = Source.Id ] // instead of scope
            forEachTarget [ // effects
                 SetIndustry value = Value + 5
                 [[MESSAGE_UNLOAD_SHIP_ON(Source.Id, Target.Id)]]
            ]
            ifTargets [ [[OTHER_TARGETS]] ] forEachTarget [ [[OTHER_EFFECT(Target.Id)]] ] // reuses activation condition 
        ]
    ]    
This is by thinking exclusively the way FOCS works today, mostly changing parse tokens.

Rethinking FOCS probably would yield a more generalized and streamlined language. E.g. free mixture of activation and targets. Either making things like ship parts first class objects or treating all meters similar in FOCS code.

Still, automated testability would be much higher on my priority list.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Config files, FOCS, Python & Co

#9 Post by Geoff the Medio »

The comment about "scope" vs. "targets" is reasonable. Several of the first few conceptual points would be good to include in a tutorial / FAQ if not already mentioned.
Ophiuchus wrote:4a) bundling of effects on different scope is not possible (e.g. i wanted in certain circumstances: a) unload the supply ship (effect to planet industry) and b) destroy the supply ship (effect to ship) )
5) no way to reuse the set of items in scope in the effects; e.g. for applying effects to a subset
You can do both of these, by including both in the scope, and then using an If condition effect to filter the targets by their object type, and then applying the appropriate effect to each.
6) not everything is an object (so, e.g. the interface for handling specials is different)
[...]
Maybe I can convey some ideas by code.
From your code, you may be wanting/expecting a greater degree of control over the order of evaluation and execution of effects than the engine is designed to provide... (eg. explicitly nested for loop logic). That said, you can probably get some of the same result by swapping your comment and forWhatever statements... eg.

Code: Select all

effects = [ // forEachTarget ...
...making things like ship parts first class objects or treating all meters similar in FOCS code.
For things like this, FOCS reflects the internal gamestate organization and logic. Parts and specials don't exist independently of their containing object... so I question whether this is an issue with FOCS or just a complicated gamestate design requiring a complicated set of commands to manipulate. I don't think it's realistic to effectively write a compiler / abstraction layer to avoid some of that complexity (nor would switching to Python instead of FOCS have any benefit).

Ophiuchus
Programmer
Posts: 3433
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Config files, FOCS, Python & Co

#10 Post by Ophiuchus »

@geoff i'm not really suggesting language changes at this point and also not suggesting switching to python (so probably by now the posts dont fit the thread topic anymore). You asked why i think that FOCS could have been done better and i tried to find a structure which would have helped me.
And what I came up with was basically
  • different naming
  • more specific naming
  • more intuitive/less complex concepts (e.g. if you add 4 parts of a kind on a ship, you have a single meter for all - i think this is very surprising if you dont read the cpp code)
  • more unified structure
.
And like you said, adding proper documentation in the wiki could probably help a lot.
Geoff the Medio wrote:
Ophiuchus wrote:4a) bundling of effects on different scope is not possible (e.g. i wanted in certain circumstances: a) unload the supply ship (effect to planet industry) and b) destroy the supply ship (effect to ship) )
5) no way to reuse the set of items in scope in the effects; e.g. for applying effects to a subset
You can do both of these, by including both in the scope, and then using an If condition effect to filter the targets by their object type, and then applying the appropriate effect to each.
Ah, thats helpful, thanks :)
Geoff the Medio wrote:That said, you can probably get some of the same result by swapping your comment and forWhatever statements... eg.

Code: Select all

effects = [ // forEachTarget ...
I actually meant that 'effects' does not tell the reader how often this gets executed, with a name like 'forEachTarget', that might be clearer (and sure, one could add helpful comments everywhere, but who does in real life?). And having a compiler check that you use the correct forWhatever statement in correct case is better than debugging.
  • item - source type - how often the effect gets called in relation to the source
  • Part ... ifSourceShip [activation condition] thenForEachPartOnTheShip
  • ShipHull ... ifSourceShip [activation condition] then
  • Special ... ifSourcePlanet [activation condition] then
  • ... (and so on)
more traditional FOCS version, compiler could enforce that first activation condition is of the correct object type:
  • Part ... activation = [ Ship <<other-conditions>> ]
  • Special ... activation = [ Planet <<other-conditions>> ]
Geoff the Medio wrote:
...making things like ship parts first class objects or treating all meters similar in FOCS code.
For things like this, FOCS reflects the internal gamestate organization and logic. Parts and specials don't exist independently of their containing object... so I question whether this is an issue with FOCS or just a complicated gamestate design requiring a complicated set of commands to manipulate. I don't think it's realistic to effectively write a compiler / abstraction layer to avoid some of that complexity (nor would switching to Python instead of FOCS have any benefit).
Totally agree. Making ship parts first class would probably mean changing gamestate design.
And I am at loss at the moment how one could unify meter usage so FOCS would reflect the current design better. Just the way it is I think, is not very intuitive/beginner-friendly. But people use it. So probably it is good enough, I guess.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

defaultuser
Juggernaut
Posts: 854
Joined: Wed Aug 26, 2015 6:15 pm

Re: Config files, FOCS, Python & Co

#11 Post by defaultuser »

I haven't looked at any of the code, so I'm unclear as to what the function of the FOCS is. Are these configuration files, or code equivalents (as in branching, looping, etc.)?

Ophiuchus
Programmer
Posts: 3433
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Config files, FOCS, Python & Co

#12 Post by Ophiuchus »

defaultuser wrote:I haven't looked at any of the code, so I'm unclear as to what the function of the FOCS is. Are these configuration files, or code equivalents (as in branching, looping, etc.)?
FOCS is for scripting of contents, a language for talking about gamestate (ships, planets...)
They are 'a little' code equivalent.
E.g. there are If tokens. Note that these ifs get executed later, not when the FOCS file gets parsed.
If sticking to the current way, when switching to python those If commands would probably not be python statements but construct If-objects.
Probably one would use normal python execution for metaprogramming (i.e. it would provide a more powerful substitute for the macro facility of FOCS).
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

defaultuser
Juggernaut
Posts: 854
Joined: Wed Aug 26, 2015 6:15 pm

Re: Config files, FOCS, Python & Co

#13 Post by defaultuser »

I see, thanks. If it were straight configuration, I would suggest XML. While that tends to be rather verbose, it's a common solution and relatively easy to pick up for developers. XML parsers are extremely common.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Config files, FOCS, Python & Co

#14 Post by adrian_broher »

XML
No.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Config files, FOCS, Python & Co

#15 Post by Vezzra »

defaultuser wrote:I see, thanks. If it were straight configuration, I would suggest XML.
Actually, IIRC game content originally had been defined by XML, before the current custom script language replaced that solution. This happened long before I joined the project, so I can only guess, but it's probably a save bet to assume XML proved to be rather unusable for that purpose.

No one wants to return to that... ;)

Post Reply