Feedback from another OSS-dev (ufoai.org)

Describe your experience with the latest version of FreeOrion to help us improve it.
Forum rules
Always mention the exact version of FreeOrion you are testing.

When reporting an issue regarding the AI, if possible provide the relevant AI log file and a save game file that demonstrates the issue.
Message
Author
User avatar
Duke
Krill Swarm
Posts: 10
Joined: Thu Apr 07, 2016 9:26 pm

Re: Feedback from another OSS-dev (ufoai.org)

#16 Post by Duke » Sun Apr 10, 2016 8:29 am

Made an interesting test:
In the above mentioned savegame, I scrapped all my 1.72k ships. The time for a single focus change went down from 35s to ~6s.
Still a little long imho. And it's pretty hard to play w/o ships ;)

User avatar
Vezzra
Release Manager, Design
Posts: 4673
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Feedback from another OSS-dev (ufoai.org)

#17 Post by Vezzra » Sun Oct 02, 2016 5:12 pm

Sorry for getting back to this after half a year, but RL got in the way, after that the release, and only now I got a few moments to dig through old, unfinished discussions.

Don't know if you're still here and listening, but in case you are, here a very delayed reponse...
Duke wrote:
Vezzra wrote:All those effects are scripted in the content scripts
I hope those scripts are somehow precompiled at startup/load time ?
Yes.
Unfortunately I don't have the time atm to set up a full IDE for FO and get familiar enough with the code to make qualified suggestions. But if you can give me handful of important filenames (where the n:n handling happens), I will take a glance or two at github.
Huh, unfortunately I'm not familiar with that part of the codebase, one of the other (more knowledgeable) devs need to chime in here. AFAIK the Condition.h/cpp and Effect.h/cpp files in the universe folder should contain the code sections you're looking for, but I likely am missing something...
btw if we are talking about design and code, are we still in the right forum ?
Once we go into more detail, we should probably open a thread in the programming subforum.

xlightwavex
Space Kraken
Posts: 111
Joined: Mon Nov 16, 2015 5:57 am

Re: Feedback from another OSS-dev (ufoai.org)

#18 Post by xlightwavex » Tue Oct 04, 2016 3:17 am

Vezzra wrote:Unfortunately, it's also extremely difficult to make improvements to this issue. There have been several attempts at such improvements...

The fundamental problem, as far as I understand, is on the design level of the effects system.
Each object in the game can have effects attached to it, and those effects can potentially affect each object.
Techs also have effects attached which also can affect each game object.
Game objects are ships, fleets, buildings, planets, systems. All those effects are scripted in the content scripts...
How is the problem of circular logic avoided then ? A effects B and if B is effected it increases its effect on A which then increases A's effect on B and so on. Priority ?
...However, each "effectsgroup" needs to evaluate its "scope" (that's the set of object it will affect) each time it's executed, and this scope always starts with the entire set of all objects.
Does this mean if we have less effect groups overall it is faster ?
Does this also mean that refactoring pre-existing scripts to be more efficient is basically equal to a speed gain.
Which is why the order of conditions for scopes in the content scripts matters a lot, because some conditions evaluate a lot faster than others, and you'd want to reduce the "result set" of a scope condition with fast conditions before the more expensive ones get applied.

So this is the idea of Implicitly trimming down the affected scope of a large set of objects.
Instead of Explicitly adding what is to be affected by adding it to a empty scope ?
Each time you e.g. change the focus of a colony, all the effects of all objects need to be re-evaluated, because there is no way to safely exclude anything for which you can safely assume that it won't be affected (directly or indirectly) by a focus change.

This will probably also tell you how difficult it is to make the effects execution implementation multi-threaded (basically, you'll have to set up an execution graph first, with all the interdependencies resolved).
I've often wondered if there is a way to redesign the fundamental effects system in a way that would allow for more efficient optimization, but so far I've been coming up blank. I'm not the most proficient coder around here, though.

User avatar
MatGB
Creative Contributor
Posts: 3310
Joined: Fri Jun 28, 2013 11:45 pm

Re: Feedback from another OSS-dev (ufoai.org)

#19 Post by MatGB » Tue Oct 04, 2016 3:51 am

Note: my answers are my current understanding, I'm still not great at this and thus I may be wrong, please correct me if you know better, but answering will help me understand it better.
xlightwavex wrote: How is the problem of circular logic avoided then ? A effects B and if B is effected it increases its effect on A which then increases A's effect on B and so on. Priority ?
Multiple passes working out what affects what: you can see this with Sitreps, which sitreps will trigger is worked out on an early pass whereas the actual effects later on, so sometimes a sitrep tells you something slightly off (eg: a different ship lost its fuel from a dampening cloud, a drydock gave a partial repair but the damage control techs had already fixed it fully, a drydock repaired ship X but the stargate had moved it to location Y where there was no drydock).

Priority is a recent introduction that we still haven't applied to everything but it makes it a lot easier to control what's actually going to happen.
Does this mean if we have less effect groups overall it is faster ?
Not by a huge amount per group, but yes, if you create your own default folder with almost nothing in it the game will play faster.
Does this also mean that refactoring pre-existing scripts to be more efficient is basically equal to a speed gain.
Yes, this is an ongoing project (see priority being new, above), and it sometimes takes someone to notice a script being inefficient and tweaking it. Sometimes the actual change was for a completely different reason but ended up massively improving efficiency, eg: we recently reworked and relocated the population bonus scripts using the priority system, the objective was that the icons would show properly in the tech window. While doing it, it was noticeable that the game sped up substantially and it was due to the effects not being in the species folder (which is used a lot for virtually every effect) but now into their source location which isn't processed anywhere near as much. After working this out we moved as many other effects as possible out and late/large game lag has been massively reduced.
So this is the idea of Implicitly trimming down the affected scope of a large set of objects.
Instead of Explicitly adding what is to be affected by adding it to a empty scope ?
Beyond me, someone?
Mat Bowles

Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12277
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Feedback from another OSS-dev (ufoai.org)

#20 Post by Geoff the Medio » Tue Oct 04, 2016 8:25 am

Evaluating the scope of an effectsgroup means finding which objects in the universe match the scope condition. In most cases, this involves applying a series of conditions to narrow down an initial list of candidate objects, within an And condition at the top level, with several other conditions contained in it.

In all cases, when evaluating a condition, some initial set of objects to consider is required. By default, or if nothing more specific has been implemented, this can be the whole universe. But when possible, the initial set is more specific: If a condition is just "Ship" then, it will start with just ship objects. If it's "InSystem id = Source.SystemID", it will look up the system object with the specified ID, then add all the objects contained by that system to the initial set. If it's "Source", it just adds the source object.

After getting the initial candidates set, all the conditions within an And condition are applied in order to those candidates, to narrow down the matches. Depending on the order they are scripted, this can be faster or slower. If you had an effect in a species script that should only work on the source ship, you could write "scope = And [ Ship Source ]" or "scope = And [ Source Ship ]". The latter should be a lot faster, because it just checks one object - the source - to see if it's a ship, rather than checking all ships to see if they are the source.

User avatar
em3
Vacuum Dragon
Posts: 510
Joined: Sun Sep 25, 2011 2:51 pm

Re: Feedback from another OSS-dev (ufoai.org)

#21 Post by em3 » Tue Oct 04, 2016 1:47 pm

This begs the question, shouldn't the scope be (stable) sorted after parsing to improve efficiency? At least the "set Source as the first condition in an And" optimization.
[...] for Man has earned his right to hold this planet against all comers, by virtue of occasionally producing someone totally batshit insane. - Randall Munroe, title text to xkcd #556

User avatar
Vezzra
Release Manager, Design
Posts: 4673
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Feedback from another OSS-dev (ufoai.org)

#22 Post by Vezzra » Tue Oct 04, 2016 1:49 pm

xlightwavex wrote:How is the problem of circular logic avoided then ? A effects B and if B is effected it increases its effect on A which then increases A's effect on B and so on.
AFAIK all the scope conditions of all effectsgroups (which determine the set of universe objects that will be affected by a effectsgroup) are evaluated before any of the effects of all the effectsgroups are executed. Which means that if the effects of effectgroup A change which objects will match the scope condition of effectsgroup B this doesn't matter, as by the time the effects of effectsgroup A are executed/applied, the scope condition of effectsgroup B has already been evaluated and the set of object effectsgroup B will applied to determined.

However, that only applies to the effects execution of a single turn. When considering multiple turns, such circular effects are not avoided. So you can implement an effectsgroup that applies changes to an universe object A, which in turn will "increase" (in whatever form) the effects of another effectsgroup that affects an universe object B. And have the changes to B "increase" the effects of our first effectsgroup.
Does this mean if we have less effect groups overall it is faster ?
Potentially. Depends on what you mean by having less effectsgroups. The number of effectsgroups in the content script isn't the factor here, but the number of effectsgroups that will be executed. One effectsgroup can be attached to a lot of objects, so that means if you have coded p ten effectsgroups in your content scripts, but each of them is only attached to one universe object, or one tech, and each one of them will therefore only be executed once each turn, you'll need less effects processing than if you only have one effectsgroup, but this one attached to, lets say 100 universe objects (because this same effectsgroup will have to be executed 100 times).

Of course this simple example assumes that all these effectsgroup need roughly the same time to be processed, but I think you get the picture.
Instead of Explicitly adding what is to be affected by adding it to a empty scope ?
As Geoff already pointed out, you need to determine a initial set to apply "and" conditions, which is what you usually have: you want to apply some effects to e.g. all objects that are ships AND in a certain system AND belong to a certain empire.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12277
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Feedback from another OSS-dev (ufoai.org)

#23 Post by Geoff the Medio » Tue Oct 04, 2016 2:06 pm

em3 wrote:...shouldn't the scope be (stable) sorted after parsing to improve efficiency? At least the "set Source as the first condition in an And" optimization.
There are few cases where this is actually done / needed. It's more of a useful example than a practical problem.

xlightwavex
Space Kraken
Posts: 111
Joined: Mon Nov 16, 2015 5:57 am

Re: Feedback from another OSS-dev (ufoai.org)

#24 Post by xlightwavex » Wed Oct 05, 2016 12:45 pm

as by the time the effects of effectsgroup A are executed/applied, the scope condition of effectsgroup B has already been evaluated and the set of object effectsgroup B will applied to determined.
I was more so asking about the order of effects execution when speaking of non commutative effects.

e.g.
you have 1 planet with 5 industry total for your empire.

Effect A adds +5 industry to all planets then 5+5 = 10
Effect B multiplies industry by 2

5+5 * 2 = 20
but in opposite order 5 * 2 + 5 = 15.

Priority then controls when a effect is executed im guessing ?
you could write "scope = And [ Ship Source ]" or "scope = And [ Source Ship ]". The latter should be a lot faster, because it just checks one object - the source - to see if it's a ship, rather than checking all ships to see if they are the source.
so

InSystem id = Source.SystemID
ship

Is better late game when there is a lot of ships then.

ship
InSystem id = Source.SystemID

Still a little confused though on how things are processed internally like, if the ships check for effects or if the effects check the ships to say what is looped thru as a executing list each turn. Objects or Effects or both.

User avatar
Vezzra
Release Manager, Design
Posts: 4673
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Feedback from another OSS-dev (ufoai.org)

#25 Post by Vezzra » Wed Oct 05, 2016 1:14 pm

xlightwavex wrote:Priority then controls when a effect is executed im guessing ?
Exactly.
Still a little confused though on how things are processed internally like, if the ships check for effects or if the effects check the ships to say what is looped thru as a executing list each turn. Objects or Effects or both.
AFAIK effects check the ships, so to speak. If I understand correctly, internally the condition "Ship" will initialize your result set with all the ships in the universe. As such a list is maintained internally, that's a very quick operation. Same applies to all universe object "types" (planets, buildings, etc.).

Post Reply