RFC: A single source for default values (FOCS, python, stringtables, backend)

For what's not in 'Top Priority Game Design'. Post your ideas, visions, suggestions for the game, rules, modifications, etc.

Moderator: Oberlus

Message
Author
User avatar
Oberlus
Cosmic Dragon
Posts: 3186
Joined: Mon Apr 10, 2017 4:25 pm

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#16 Post by Oberlus »

Oh... I misunderstood.

Then, what would be the point of this?:
Geoff the Medio wrote: Fri May 29, 2020 3:26 pm
If I read you correctly you would expect the following to work/be usable in stringtables?

MACRO_OUTER
'''[[MACRO_SPACE]]'''

MACRO_SPACE
''' '''
yes

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#17 Post by Ophiuchus »

Ophiuchus wrote: Fri May 29, 2020 10:23 am
Oberlus wrote: Fri May 29, 2020 9:31 amWhat I am doing wrong?
Nothing, I think it is not supported. At least not for calculated macros.
The thing I asked geoff about was if macros get expanded recursively. Does not give access to game rule values or multipliers though.

I did a web survey to find a language which supports what we need to bring FOCS, stringtables, backend and AI together.

I found a lot of configuration languages like TOML, YAML etc which improve on JSON and reduce boilerplate by providing some merging/reuse of definitions. This is nice for but does not solve the issues at hand and for many things we can use macros on an outer layer. Also those often have good support for dates/timezones and units (like s,kg,MB) which are important for system configuration but we do not really need.

YAML supports references (anchors) not sure if that is helpful.

Most others like HOCON and CUE are rather single-tool/platform (JVM and go).

The closest languages (configuration languages with arithmetics support) I found are CUE/HCL/GCL/jsonnet.
CUE and HCL are single-tool (both go-based). GCL (by google) is not public at all. That there is not much tool support might be due to the reason that these are not so minimal anymore, some are even turing-complete, so it is harder to implement support in multiple platforms.

jsonnet is a superset of JSON (many are familiar with that), written in C++ (second implementation in go), has bindings for C,python,and C++ (this would fit the bill). can also be crosscompiled to js, also some bindings for six other languages by third partys. I am not sure the template approach it follows is a way to go. Also it does not provide validation. It addresses the usual JSON suspects (multiline strings, comments, allow dangling commas,...) . It allows for arithmetics. It also has a lot of stuff we do not really need.

CUE lang configuration use case has some comparisons with jsonnet, GCL and HCL (if you scroll down to "Comparisons") and also hightlights the CUE benefits.

I imagine this could work, first macro expansion, then running the jsonnet interpreter priming with game rule values, then processing content definitions, feeding that to the FOCS parser and for human player also process stringtables. Dumping to filesystem seems like the wrong thing to do so just keep this in memory and fetch stringtables from there. Interpreter needs to be run on game start, on opening encyclopedia on start screen (with defaults for the game values), or when switching stringtables.

Probably more sensible to split simple-arithmetic-properties to its own definition files usable from stringtables and AI (should make AIDependencies mostly obsolete). and still to split calculated values from current FOCS and a way to use the resulting values in macro expansion.

Anyway jsonnet is much too huge for inclusion (60MB or sth).

Need to think about this a bit more.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

o01eg
Programmer
Posts: 981
Joined: Sat Dec 10, 2011 5:46 am

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#18 Post by o01eg »

Ophiuchus wrote: Sat May 30, 2020 5:09 pm The thing I asked geoff about was if macros get expanded recursively. Does not give access to game rule values or multipliers though.

I did a web survey to find a language which supports what we need to bring FOCS, stringtables, backend and AI together.

I found a lot of configuration languages like TOML, YAML etc which improve on JSON and reduce boilerplate by providing some merging/reuse of definitions. This is nice for but does not solve the issues at hand and for many things we can use macros on an outer layer. Also those often have good support for dates/timezones and units (like s,kg,MB) which are important for system configuration but we do not really need.

YAML supports references (anchors) not sure if that is helpful.

Most others like HOCON and CUE are rather single-tool/platform (JVM and go).

The closest languages (configuration languages with arithmetics support) I found are CUE/HCL/GCL/jsonnet.
CUE and HCL are single-tool (both go-based). GCL (by google) is not public at all. That there is not much tool support might be due to the reason that these are not so minimal anymore, some are even turing-complete, so it is harder to implement support in multiple platforms.

jsonnet is a superset of JSON (many are familiar with that), written in C++ (second implementation in go), has bindings for C,python,and C++ (this would fit the bill). can also be crosscompiled to js, also some bindings for six other languages by third partys. I am not sure the template approach it follows is a way to go. Also it does not provide validation. It addresses the usual JSON suspects (multiline strings, comments, allow dangling commas,...) . It allows for arithmetics. It also has a lot of stuff we do not really need.

CUE lang configuration use case has some comparisons with jsonnet, GCL and HCL (if you scroll down to "Comparisons") and also hightlights the CUE benefits.

I imagine this could work, first macro expansion, then running the jsonnet interpreter priming with game rule values, then processing content definitions, feeding that to the FOCS parser and for human player also process stringtables. Dumping to filesystem seems like the wrong thing to do so just keep this in memory and fetch stringtables from there. Interpreter needs to be run on game start, on opening encyclopedia on start screen (with defaults for the game values), or when switching stringtables.

Probably more sensible to split simple-arithmetic-properties to its own definition files usable from stringtables and AI (should make AIDependencies mostly obsolete). and still to split calculated values from current FOCS and a way to use the resulting values in macro expansion.

Anyway jsonnet is much too huge for inclusion (60MB or sth).

Need to think about this a bit more.
What about to write configuration in python and parse it with ast module?
Gentoo Linux x64, gcc-9.3, boost-1.74.0
Ubuntu Server 18.04 x64, gcc-7.4, boost-1.65.1
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 0.4.10.1.
Donations are welcome: BTC:14XLekD9ifwqLtZX4iteepvbLQNYVG87zK

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#19 Post by Ophiuchus »

o01eg wrote: Sat May 30, 2020 6:33 pmWhat about to write configuration in python and parse it with ast module?
What do you propose exactly? Also would this be usable from c++? You can answer that in the original python thread Potential replacement of FOCS with Python.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12802
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#20 Post by Geoff the Medio »

o01eg wrote: Sat May 30, 2020 6:33 pmWhat about to write configuration in python and parse it with ast module?
I don't really understand the point of this suggestion or the various other scripting or serialization formats that have been suggested in the same context. What is the potential benefit, in terms of accessing data in different places, of rewriting FOCS content in another format (ignoring any other benefits, like easy of writing scripts or availability of tools to do so)? If a building type is scripted in FOCS or Python or JSON or YAML or whatever else, or some combination, how does that make it easier to access from a stringtable entry? Is it also suggested to write stringtables and AI in YAML somehow, and if so, why?

Accessing scripted content info via Python is already possible through the Python API, which exposes stuff like ShipParts, Species, or Specials and various properties. Macros can insert text into FOCS content script files or stringtable entries from a common source. Changing scripting of content to YAML wouldn't change that. Possibly if content was scripted as Python it might be easier to access the content from Python, but not from stringtables.

What seems to be missing is a way to lazy evaluate FOCS expressions within stringtable entries, when those stringtable entries are displayed. This would need to work similar to text formatting tags like <i> and <u>, images like "<img src="species/scylior.png"></img>" or links like [[encyclopedia ORGANIC_SPECIES_TITLE]] (which is replaced by the stringtable loader with <encyclopedia ORGANIC_SPECIES_TITLE>Organic Metabolism</encyclopedia>), with the formatting markup in the stringtable being replaced or interpreted to control how text is shown in suitable GUI widgets, like a pedia entry or tooltip (and not when the text is read in like a [[MACRO]]).

A feature like this already exists in the stringtables, though in a limited fashion: Various stringtable entries have placeholders like %1% and %empire%, which are replaced when the text is shown in the UI by context-appropriate values. The values available for such substitution are limited by the context in which a stringtable entry is used, though, and not freely configurable.

So perhaps all or most of what is needed is a way to embed a FOCS expression into a stringtable, to be evaluated when the text is displayed or when the relevant GUI widget is initialized. This would be useful to look up values of things like the fuel or speed of a hull, or the homeworld of a species, or what turn it is in a game. It could potentially also evaluate arbitrary FOCS expressions, like doing calculations on those numbers, or even running statistic with conditions and accumulated object property values on the game universe and converting the result back to text.

I'd imagine a format for embedding FOCS in stringtable to be something like <FOCS>ShipDesignsOwned empire = %1%</FOCS>, where everything within the <FOCS> tags would be expanded, substituted with any use-context dependent values like %1% for an empire ID, then evaluated as a FOCS ValueRef, then converted to text, and inserted in place of the <FOCS> tags and their contents. So if this was used in a pedia article about an empire, then the empire ID would be known a passed as in for replacing %1%.

This would be useful for pedia articles like

Code: Select all

ENC_GALAXY_SETUP_SETTINGS
'''Seed: %1%
Number of Systems: %2%
Shape: %3%
Age: %4%
Starlanes: %5%
Planets: %6%
Specials: %7%
Monsters: %8%
Natives: %9%
AI Max Aggression: %10%
Game UID: %11%
'''
in that several of those values could be specified as FOCS expressions instead of needing to be passed in as numbered substitutions by the GUI displaying the text.

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#21 Post by Ophiuchus »

Geoff the Medio wrote: Mon Jun 01, 2020 10:28 pm
o01eg wrote: Sat May 30, 2020 6:33 pmWhat about to write configuration in python and parse it with ast module?
I don't really understand the point of this suggestion or the various other scripting or serialization formats that have been suggested in the same context. What is the potential benefit, in terms of accessing data in different places, of rewriting FOCS content in another format (ignoring any other benefits, like easy of writing scripts or availability of tools to do so)? If a building type is scripted in FOCS or Python or JSON or YAML or whatever else, or some combination, how does that make it easier to access from a stringtable entry? Is it also suggested to write stringtables and AI in YAML somehow, and if so, why?
+1

I think there is not really a connection between string tables and the content scripting language.
Geoff the Medio wrote: Mon Jun 01, 2020 10:28 pm What seems to be missing is a way to lazy evaluate FOCS expressions within stringtable entries, when those stringtable entries are displayed.
Yes, that would work for stringtables in most cases I guess. For looking up properties like hull structure etc we may be missing the right context (but that is extended use cases anyway). But in order to have a single source of truth for maintenance in many cases it will still be necessary to have macros - which is fine for stringtables as we do macro expansion there. But in that cases it wont work for AI (currently).

So I think in more general terms we need some nominal lookup (instead or as a variant of macro expansion) of calculated values.

I can think of three variants
  1. macro expansion and FOCS tags for stringtables; macro expansion and FOCS parsing for backend; macro text extraction and generic FOCS calc API (FOCS-string to value) for python AI
  2. macro expansion and nominal tags for stringtables; macro expansion and FOCS parsing for backend; nominal value API (name-string to value) for python AI
  3. macro expansion and python/FOCS/nominal tags for stringtables; game content construction API for backend; python readable content
I think the first variant without AI support is probably the easiest to achieve (in a simple form) and gives the best short term gain (calculated stringtables values based on invariant values). One could stringtable damage/structure gains of weapon tech upgrade effects based on GameRules, and maybe even shippart gains based on current tech (e.g. fuel tank), also better sitreps.

If a more complex form is achievable the best thing about generic FOCS (or similar) tags would be that one could also use it for source object variant values. Not sure how this works though (somewhere there would have to be given some context object).

Macro text extraction would allow some kind of pseudo-nominal lookup. python would need to learn how to parse FOCS files for macros (name and text) and for well-known names it would send the corresponding FOCS snippet to the lookup/eval api, which would parse and evaluate the snippet. For that we would probably reuse the ValueRef parser, but this might be fiddly. Another variant: the AI coude could learn how to interpret the small necessary subset of FOCS itself, but if even only evaluating a subset, I think it makes more sense to keep this in the server.

Nominal tags should be kind of easy. Add a single-function lookup API. Add something like a "Named" valueRef in FOCS

Code: Select all

Named name = "REINFORCED_HULL_BONUS" value = [[SHIP_STRUCTURE_FACTOR]]  * 5
and use that in the stringtables

Code: Select all

The reinforced hull tech adds [[Value REINFORCED_HULL_BONUS]] to maximum structure of each empire ship
and AI

Code: Select all

fo.getNamedValue("REINFORCED_HULL_BONUS")

The third variant moves content construction to python, which gives access to powerful scripting (e.g. generating a tech cascade via a loop) and direct access to content structure for the AI. But it is also quite some work (although i think all the content should already have bindings written for the querying part). GameRule values are/can be probably passed to AI on startup so it can be made trivially aware of calculated values. Maybe also some default understanding of simple effects.
Also having a content construction AI gives more flexibility - one could add content for testing on the fly, e.g. directly in the debug chat.
So I think this has the highest payoff for scripting but could also be the most work. It still needs an orthogonal solution to stringtables - either one of the mentioned variants or setting nominal values via API like

Code: Select all

fo.setStringtableValue("REINFORCED_HULL_BONUS", calculated_bonus)
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#22 Post by Ophiuchus »

I started with implementing variant 2 (nominal referencing) in PR-3000.

I did the parser and ValueRef work for FOCS (e.g. Named name = "BLABLA" value = 3 * 3), but now i need:
  • a place to store the name value pairs
  • a way to look up the value when processing stringtable entries
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#23 Post by Ophiuchus »

Ophiuchus wrote: Tue Jun 02, 2020 10:21 pm I started with implementing variant 2 (nominal referencing) in PR-3000.
Discussion from there:
geoffthemedio wrote: "Naming" FOCS ValueRef expressions or higher-level things (like effectsgroups) was considered, but is unnecessary. The macro mechanism accomplishes the same thing, but much more flexibly.
Agreed, having a naming facility in contexts where macro mechanism/names and ValueRef evaluation are available is unnessecary.

But this is neither true for stringtables (no ValueRef evaluation), nor for python AI (no macro support at all) as there are no ids or names for value refs registered in the backend.

One could reuse the macro definitions but that has some side issue:
  • currently after macro expansion the names of the text snippets are gone
  • macro names are not guaranteed to be unique
  • also for nested macros, a macro might expand to multiple implementations.
  • macros contain arbitrary text snippets, not valueRefs or someting parseable by itself
geoffthemedio wrote: Aside from that, there's no need for a "named valueref" to be a type of valueref. There would just need to be a list of valuerefs with names associated.
Yes, true. For marking a valueref (like the one in the PR) there is still a nesting toking in FOCS necessary. Probably could do the hard work in the parser.

Also note for the scope of this issue I try to look up a value and not a (lazy) valueRef, so the idea would be to check for invariance (else log a warning), then eval the value ValueRef and register the resulting value. It might be that looking up the ValueRef instead makes more sense but i did not ponder lifetime issues yet.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#24 Post by Ophiuchus »

I yesterday also tried to find a good place to hook into stringtable processing, but could not really find one.

Both [Eval [[FOCS-VALUEREF]] ] and [Value VALUE-NAME] are different from the facilities we have.
In UI/LinkText links are created, but content does not get substituted.

In GG/Font italics and similar tags are processed.

I did not find the image part yet.

Not sure what to do. Should I add it to the LinkText? Maybe hook it to the CUILinkTextMultiEdit via multi inheritance similar to the TextLinker?
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12802
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#25 Post by Geoff the Medio »

Ophiuchus wrote: Wed Jun 03, 2020 7:37 amYes, true. For marking a valueref (like the one in the PR) there is still a nesting toking in FOCS necessary. Probably could do the hard work in the parser.
There's not much difference in the parser, I think... Whether or not there is a "Named" ValueRef derived class, you'd need to parse a list of (or perhaps randomly scattered) name+ValueRef expression pairs, and then put them into some container, indexed by the name.
Also note for the scope of this issue I try to look up a value and not a (lazy) valueRef, so the idea would be to check for invariance (else log a warning), then eval the value ValueRef and register the resulting value. It might be that looking up the ValueRef instead makes more sense but i did not ponder lifetime issues yet.
Invariances probably aren't enough; I think you'd need to restrict to constant-valued expressions with no gamestate dependence. A count of how many ships exist in the universe is invariant to source, target, localcandidate, and rootcandidate, but will not work properly if evaluated just once, especially if the just once is before a game is being played. And they probably need to have no dependence on other parsed content either, as it might otherwise depend on what scripted content has been loaded, which can potentially change on the client after execution starts. These issues could be avoided if you store the ValueRefs, not their results, and Eval() them when needed. They need to be parsed and constructed to be evaluated anyway, so I don't think storing the actual ValueRef, rather than its Eval() result is much more difficult / complicated. And constant ValueRefs already cache their result, so you caching the result externally doesn't really optimize or otherwise have much benefit...

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12802
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#26 Post by Geoff the Medio »

Both [Eval [[FOCS-VALUEREF]] ] and [Value VALUE-NAME] are different from the facilities we have.
In UI/LinkText links are created, but content does not get substituted.
LinkText does substitute the names of objects, like ships or systems, which is quite similar to the functionality you want.
In GG/Font italics and similar tags are processed.
GG code is probably too low level. It doesn't know anything about ValueRefs or the gamestate or scripted content or similar.
I did not find the image part yet.
Images in pedia articles are handled by GG/RichText.
Not sure what to do. Should I add it to the LinkText? Maybe hook it to the CUILinkTextMultiEdit via multi inheritance similar to the TextLinker?
In LinkText would make sense, perhaps as a LinkDecorator. Doesn't need to actually be a clickable link or to have any special decoration / colouration, so I'm not sure if there would be quirks or problems with using that mechanism for value substitutions, but it can probably do something similar and that should then work in sitreps and pedia text.

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#27 Post by Ophiuchus »

The PR has a working implementation (for encyclopedia and sitreps). Making the implementation right needs more work. UI needs more work. And python interface for AI is not there yet.

The main motivation for doing this now was the addition of game rule based scaling to bouts, damage and structure values which makes the static stringtable contents even more outdated.

Content I currently cover fuel tech and parts and the reinforced hull effect.

Who/Anybody can support me on the python API/bridging? I access the registered value refs via GetValueRef(name) (defined in ValueRefs.cpp). The returned untyped valueref has a StringResult() method. Probably exposing this via a GetStringResult(name) function would make sense - maybe that is all python needs. Or maybe rather a GetNumericalResult(name) function as we want to use this (e.g. effective bonus of reinforced hull tech) in computations.

I need some help on the UI. So anybody in for testing (no technical experience required)? I could build a snap if that helps.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#28 Post by Ophiuchus »

Implementation is almost right. It seems not to work on windows (geoff showed a non-working screenshot), so it would be good if somebody else could have a look on windows or/and macos (have only linux dev machines).

And bump for asking about help for AI and UI:
Ophiuchus wrote: Thu Jun 18, 2020 12:49 pm Who/Anybody can support me on the python API/bridging? I access the registered value refs via GetValueRef(name) (defined in ValueRefs.cpp). The returned untyped valueref has a StringResult() method. Probably exposing this via a GetStringResult(name) function would make sense - maybe that is all python needs. Or maybe rather a GetNumericalResult(name) function as we want to use this (e.g. effective bonus of reinforced hull tech) in computations.

I need some help on the UI. So anybody in for testing (no technical experience required)? I could build a snap if that helps.
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
Oberlus
Cosmic Dragon
Posts: 3186
Joined: Mon Apr 10, 2017 4:25 pm

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#29 Post by Oberlus »

Ophiuchus wrote: Thu Jun 18, 2020 12:49 pm I need some help on the UI. So anybody in for testing (no technical experience required)? I could build a snap if that helps.
I can't build on Windows. Do you need playtesting on Linux anyways?

Ophiuchus
Programmer
Posts: 1821
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: RFC: A single source for default values (FOCS, python, stringtables, backend)

#30 Post by Ophiuchus »

Oberlus wrote: Fri Sep 04, 2020 4:11 pm
Ophiuchus wrote: Thu Jun 18, 2020 12:49 pm I need some help on the UI. So anybody in for testing (no technical experience required)? I could build a snap if that helps.
I can't build on Windows. Do you need playtesting on Linux anyways?
Yes, for the UI
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

Post Reply