Feedback from another OSS-dev (ufoai.org)

Describe your experience with the latest version of FreeOrion to help us improve it.

Moderator: Oberlus

Forum rules
Always mention the exact version of FreeOrion you are testing.

When reporting an issue regarding the AI, if possible provide the relevant AI log file and a save game file that demonstrates the issue.
Message
Author
User avatar
Duke
Krill Swarm
Posts: 10
Joined: Thu Apr 07, 2016 9:26 pm

Feedback from another OSS-dev (ufoai.org)

#1 Post by Duke »

First off, thanks for creating this great game :)

I have enjoyed playing FreeOrion occasionally for a few years now, but I always won by researching singularity. Yesterday I finished a game by conquering the galaxy for the first time (because I *wanted* it; could have got singularity 50 turns earlier).

The galaxy had 150 systems and I populated every planet. Using the official release from around sept 2015.
Starting around turn 250, two actions became a pita:
  • adding to/changing the production queue
  • changing the focus of a planet
I got freezes between 5 and 20 secs on an i7. For each an every of those actions. Increasing from turn to turn. And I played until 290.

Forum search revealed that this has been reported before (about a year ago). So my question is: has this been fixed and should I turn to bleeding edge versions ?

Oh, and I have one feature request: can we have a NOT-operator in the objects window plz ? Like in "show myEmpire planets where species is NOT human".

User avatar
Cpeosphoros
Space Kraken
Posts: 124
Joined: Sat Jan 30, 2016 11:29 am

Re: Feedback from another OSS-dev (ufoai.org)

#2 Post by Cpeosphoros »

Duke wrote: I got freezes between 5 and 20 secs on an i7. For each an every of those actions. Increasing from turn to turn. And I played until 290.

Forum search revealed that this has been reported before (about a year ago). So my question is: has this been fixed and should I turn to bleeding edge versions ?
It's way, way better with this week builds.
Duke wrote:Oh, and I have one feature request: can we have a NOT-operator in the objects window plz ? Like in "show myEmpire planets where species is NOT human".
I've just opened https://github.com/freeorion/freeorion/issues/599. I like the idea.
All contributions are released under GPL or LGPL v2 or later, or under appropriate Creative Commons licence, consistent with project guidelines.

User avatar
MatGB
Creative Contributor
Posts: 3310
Joined: Fri Jun 28, 2013 11:45 pm

Re: Feedback from another OSS-dev (ufoai.org)

#3 Post by MatGB »

Duke wrote: Forum search revealed that this has been reported before (about a year ago). So my question is: has this been fixed and should I turn to bleeding edge versions ?
Fixed? No. Massively improved? Yes. And there's an Option in the settings to theoretically improve it more but I found that annoying.
Oh, and I have one feature request: can we have a NOT-operator in the objects window plz ? Like in "show myEmpire planets where species is NOT human".
Yes please, can we someone?

Also, an easy "just show my stuff" checkbox that can be applied with other filters easily.
Mat Bowles

Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

User avatar
Bromstarzan
Dyson Forest
Posts: 206
Joined: Sun Feb 28, 2016 9:56 pm
Location: Sweden

Re: Feedback from another OSS-dev (ufoai.org)

#4 Post by Bromstarzan »

Duke wrote: I got freezes between 5 and 20 secs on an i7. For each an every of those actions. Increasing from turn to turn. And I played until 290.
I just played through a 507 turns game (build 1509, 150 systems) and did not experience any freezing at all, but I suppose it is related to number of AIs (I only used 1 AI). Perhaps also the choice of galaxy layout matters (starlane connections etc).
Duke wrote: Oh, and I have one feature request: can we have a NOT-operator in the objects window plz ? Like in "show myEmpire planets where species is NOT human".
I like this suggestion.
MatGB wrote:Also, an easy "just show my stuff" checkbox that can be applied with other filters easily.
Would be very handy, indeed!
| i7 7700K [email protected] | GTX 1080 Ti | RAM: 32GB | PSU: 750w | W10 x64 | 2xAcer1920x1080 |

User avatar
Duke
Krill Swarm
Posts: 10
Joined: Thu Apr 07, 2016 9:26 pm

Re: Feedback from another OSS-dev (ufoai.org)

#5 Post by Duke »

Bromstarzan wrote:I just played through a 507 turns game (build 1509, 150 systems) and did not experience any freezing at all,
Very interesting. Could you plz try my savegame on your machine ? Your machine is a bit stronger than the one I played on ([email protected], 12GB, Vista), but that can't result in NO freeze vs. 15 secs.
And I would like to test your savegame on my machine.
Bromstarzan wrote: ..., but I suppose it is related to number of AIs (I only used 1 AI). Perhaps also the choice of galaxy layout matters (starlane connections etc).
Number of AIs *should* not matter when changing *my* production queue imho. If it did, I would call it a 'design bug'.
Starlanes could well be the reason. Because the allocation of PP to the production items is recalculated on reorder, the code may also check the availability of the PPs in the locations, recalculating the supplylines again.
Another suspicion of mine is that all the items of all times are kept in the storage and only 'marked as deleted' when they are used.

Maybe one of the devs can enlighten us ?

User avatar
Duke
Krill Swarm
Posts: 10
Joined: Thu Apr 07, 2016 9:26 pm

Re: Feedback from another OSS-dev (ufoai.org)

#6 Post by Duke »

Seems like I can not attach my 10MB savgame here.
So what is the preferred method for exchanging savegames here ? pastebin ?

User avatar
MatGB
Creative Contributor
Posts: 3310
Joined: Fri Jun 28, 2013 11:45 pm

Re: Feedback from another OSS-dev (ufoai.org)

#7 Post by MatGB »

I'm not a dev, I'm sort of playtest/balance lead and can't really code (but can script a bit). It's mostly linked to the number of effects creating objects within the game, each colony/tech/ship/part creates work for the engine, and changing production, etc can require processing a lot of changes.

The more Objects there are, the longer the processing time—I've always tried to keep the number of ships I controlled below 200 (scrapping obsolete ships, excess scouts, etc) and try to have galaxy settings so that I'm close to winning before I have ~150 planets, however since the recent changes I've been upping my planet limit gradually and it is still not noticeably laggy.

Regarding savegame sharing, people have used pastebin and dropbox, or even simply emailing them around, zipping files can work, however...

A game generated by last Septembers Release will be completely incompatible with current install, someone would have to go back and reinstall, and you couldn't compare lag with the same specific game with different versions (policy is to try to avoid breaking savegames but prioritise making the game better, and there's been a lot changed, especially in the AI which will almost certainly simply fail).

Combine that with needing to use the larger, and slower, XML savegame format over the binary format if you're switching platforms and we just don't normally swap saves.

For what it's worth, number of AIs makes a difference, but not in the way most people think (including me), it's not the work the AI is doing that makes a difference, it's that each surviving AI will have fleets, colonies, etc and each needs to be processed. And yes, that this slows down processing in your personal production window is a problem and is, partially, fixed in Trunk. I think.

Maybe an actual dev (Geoff?) can weigh in? late game lag has annoyed me for ages, and I started playing on a netbook.
Mat Bowles

Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

User avatar
Duke
Krill Swarm
Posts: 10
Joined: Thu Apr 07, 2016 9:26 pm

Re: Feedback from another OSS-dev (ufoai.org)

#8 Post by Duke »

Ok, so the savegame swap is postponed until I played a full game on a recent version.
MatGB wrote:I've always tried to keep the number of ships I controlled below 200
That could be an important hint. As I prefer 'glass cannons' (ie. a small(organic) hull, one gun, nothing else), I have hundreds of ships. Must have been around 1.5k by turn 290.
However, I still fail to see why ships have to be processed every time I change my queue.

User avatar
Duke
Krill Swarm
Posts: 10
Joined: Thu Apr 07, 2016 9:26 pm

Re: Feedback from another OSS-dev (ufoai.org)

#9 Post by Duke »

MatGB wrote: late game lag has annoyed me for ages, ...
And may I add: This is the *only* annoying flaw I encountered in this otherwise high-quality game, so fighting this issue should be prio 1 imho.
I have certainly seen several minor glitches and things that could have been solved a little better. But a 15s lag for each action is game-breaking.

User avatar
MatGB
Creative Contributor
Posts: 3310
Joined: Fri Jun 28, 2013 11:45 pm

Re: Feedback from another OSS-dev (ufoai.org)

#10 Post by MatGB »

I basically agree entirely. One other suggestion: for focus changing, do it entirely through the Objects menu (right click a planet or ctrl-select a bunch of planets) with the system sidepanel closed. It's a lot faster and I still don't know why, it appears to not force changes of various UI states but the rest is beyond me.

On the rest, it's all, obviously, a work in progress, my main focus is on testing new features and then balancing them by tweaking numbers in the scripts, etc. I also try to do various balance passes on existing stuff to make sure that while you have choices, it's of an interesting "I want to do this today" not of a "I want to win/lose because I made the wrong choice" nature.

Oh, Fleet Upkeep—not documented as well as it should be because, well, we're changing it soon(ish), every single ship you control increases the production cost of all new ships by 1%. Lots of glass cannon small ships becomes very expensive over time compared to a smaller number of tougher ships. I was going to change upkeep to per part used but then we came up with a better plan, both will remove the massive incentive for large ships over small in the current system.

Which will also mean we need to do more to solve the lag problem for big fleets as when the change comes in I'll be doing some very silly designs using teeny ships to see what I can get away with, and I hate the lag.
Mat Bowles

Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Feedback from another OSS-dev (ufoai.org)

#11 Post by Vezzra »

Duke wrote:And may I add: This is the *only* annoying flaw I encountered in this otherwise high-quality game, so fighting this issue should be prio 1 imho.
Unfortunately, it's also extremely difficult to make improvements to this issue. There have been several attempts at such improvements, some made things at least a bit better, some failed (a few years ago one of our contributors, cami, tried to rewrite the entire effects execution system to make it more efficient and, most of all, provide a clean multi-threaded solution, but ran out of time before he had to quit).

The fundamental problem, as far as I understand, is on the design level of the effects system. Each object in the game can have effects attached to it, and those effects can potentially affect each object. Techs also have effects attached which also can affect each game object. Game objects are ships, fleets, buildings, planets, systems. All those effects are scripted in the content scripts, and by allowing each effect to potentially affect each game object, you get a lot of flexibility. But you also get a horrible performace, because we're talking about worst case O(n^2) here (n being the number of game objects). In reality it's of course much lower, because not every effect affects each object. However, each "effectsgroup" needs to evaluate its "scope" (that's the set of object it will affect) each time it's executed, and this scope always starts with the entire set of all objects. Which is why the order of conditions for scopes in the content scripts matters a lot, because some conditions evaluate a lot faster than others, and you'd want to reduce the "result set" of a scope condition with fast conditions before the more expensive ones get applied.

Each time you e.g. change the focus of a colony, all the effects of all objects need to be re-evaluated, because there is no way to safely exclude anything for which you can safely assume that it won't be affected (directly or indirectly) by a focus change.

This will probably also tell you how difficult it is to make the effects execution implementation multi-threaded (basically, you'll have to set up an execution graph first, with all the interdependencies resolved).

I've often wondered if there is a way to redesign the fundamental effects system in a way that would allow for more efficient optimization, but so far I've been coming up blank. I'm not the most proficient coder around here, though.

User avatar
Duke
Krill Swarm
Posts: 10
Joined: Thu Apr 07, 2016 9:26 pm

Re: Feedback from another OSS-dev (ufoai.org)

#12 Post by Duke »

MatGB wrote:for focus changing, do it entirely through the Objects menu
Holy shit ! What a difference...
Focus change:
1 planet via sidepanel : 35s
1 planet via objects (sidepanel open) : 35s
4 planets via objects (sidepanel open) : 2:20
4 planets via objects (sidepanel closed) : NO time ! As soon as I can open the sidepanel again, the focus has already changed.
That's a perfect workaround for the focus change :)

And it is a great hint for the devs where to look for the bug. It seems to be the notification of open windows about an update of the data and not the calculation of the data itself, because the latter also happens when the sidepanel is opened and takes only half a second maybe. Ok, pretty much a blind shot. I didn't look at the code.

Do you happen to have such a workaround for the production queue as well ?

User avatar
MatGB
Creative Contributor
Posts: 3310
Joined: Fri Jun 28, 2013 11:45 pm

Re: Feedback from another OSS-dev (ufoai.org)

#13 Post by MatGB »

I wish. I did ask for it but the guy doing it had to go do other stuff for a bit, not heard from him recently.
Mat Bowles

Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

User avatar
Duke
Krill Swarm
Posts: 10
Joined: Thu Apr 07, 2016 9:26 pm

Re: Feedback from another OSS-dev (ufoai.org)

#14 Post by Duke »

Thanks Vezzra, your explanations made things much more clear to me :)
Vezzra wrote:All those effects are scripted in the content scripts
I hope those scripts are somehow precompiled at startup/load time ?

At least such an n:n scenario offers one hope: if you can reduce the number of objects by 10%, the number of relations goes down by 19% ;) The key to an n:n problem is usually grouping. Like in "if the fleet is out of range of the effect, all ships in the fleet are too".
And a star system could be seen as a group of planets and their buildings. But you guys have probably already done that and I will stop this wild guessing now until I looked at the code.

Unfortunately I don't have the time atm to set up a full IDE for FO and get familiar enough with the code to make qualified suggestions. But if you can give me handful of important filenames (where the n:n handling happens), I will take a glance or two at github.

btw if we are talking about design and code, are we still in the right forum ?

SkyCore
Space Floater
Posts: 23
Joined: Tue Apr 05, 2016 4:37 pm

Re: Feedback from another OSS-dev (ufoai.org)

#15 Post by SkyCore »

Vezzra wrote:
Duke wrote:And may I add: This is the *only* annoying flaw I encountered in this otherwise high-quality game, so fighting this issue should be prio 1 imho.
Unfortunately, it's also extremely difficult to make improvements to this issue. There have been several attempts at such improvements, some made things at least a bit better, some failed (a few years ago one of our contributors, cami, tried to rewrite the entire effects execution system to make it more efficient and, most of all, provide a clean multi-threaded solution, but ran out of time before he had to quit).

The fundamental problem, as far as I understand, is on the design level of the effects system. Each object in the game can have effects attached to it, and those effects can potentially affect each object. Techs also have effects attached which also can affect each game object. Game objects are ships, fleets, buildings, planets, systems. All those effects are scripted in the content scripts, and by allowing each effect to potentially affect each game object, you get a lot of flexibility. But you also get a horrible performace, because we're talking about worst case O(n^2) here (n being the number of game objects). In reality it's of course much lower, because not every effect affects each object. However, each "effectsgroup" needs to evaluate its "scope" (that's the set of object it will affect) each time it's executed, and this scope always starts with the entire set of all objects. Which is why the order of conditions for scopes in the content scripts matters a lot, because some conditions evaluate a lot faster than others, and you'd want to reduce the "result set" of a scope condition with fast conditions before the more expensive ones get applied.

Each time you e.g. change the focus of a colony, all the effects of all objects need to be re-evaluated, because there is no way to safely exclude anything for which you can safely assume that it won't be affected (directly or indirectly) by a focus change.

This will probably also tell you how difficult it is to make the effects execution implementation multi-threaded (basically, you'll have to set up an execution graph first, with all the interdependencies resolved).

I've often wondered if there is a way to redesign the fundamental effects system in a way that would allow for more efficient optimization, but so far I've been coming up blank. I'm not the most proficient coder around here, though.
How about a 2-d matrix which contain pointers to the game objects ordered such that their physical location corresponds to their position on the game field. It doesnt need to be an exact 1-to-1, you could have several dozen au match a single node in the matrix. Now when your effects need to be checked for you have a very small subset of all game objects to sift through. Just check the matching matrix node, and potentially adjacent nodes if your effect has a larger radius than is fully contained. Should be orders of magnitude quicker.

//2d matrix of the following structure
int *head; //address of the list of pointers
int capacity; //current memory allocated for this node
int numElements; //number of pointers currently held for this node

Post Reply