heavy client slowdown on large games

[bigeagle]

After two testgames i was curious where the scale limits of freorion are as i really like large scale games. So i just maxed out the settings
5000 systems
20 AI
low starlane, monster, planet, natives density

And i was positively surprised that it ran very well until about turn 200+ the slowdown became more and more serious. Atm i'm at turn 280 and it takes about 5 seconds to enqueue some building, the game freezes until it is done.
Also typing names for ships is quite hard if the input is that slow. Especially the first keypress is doubled, quickly pressed keys after that appear 'solo', just very slow.
I find this sad because it's the only problem until now. The game runs quite stable, turn times are ok (in my opinion at least) and i'm fascinated to see a really parallel AI.

Version: 0.4.4 [SVN 7641] MSVC 2010 (the default windows release on sourceforge atm)
on win7 64bit, i5 [email protected] GHz and 16 GB RAM, so it should'nt be a hardware issue. It makes nearly no difference if background tasks are running (BOINC). FPS on the galaxy map are about 18 zoomed in, 60 zoomed out. No difference for the freezetime to add something to the buildqueue.

I'm curious why. Is the problem known? Maybe already fixed?
My first thought was that there must be a linear search over an array of all colonies (including AI) or something but i doubt that it is that easy. The code seems to be quite complex (at least for a c++ beginner like me) so i didn't find anything useful on my lazy/leisure search.

i just stumpled over the logfile as i looked how big the savegame is
--- snip ---
2015-05-11 14:52:15,580 DEBUG Client : ========= Production Update for empire: 1 ========
2015-05-11 14:52:15,657 DEBUG Client : ProductionQueue::Update: Simulating future turns of production queue
2015-05-11 14:52:15,726 DEBUG Client : ProductionQueue::Update: Projections took 23003 microseconds with 8307.16 total Production Points
2015-05-11 14:52:15,726 DEBUG Client : MapWnd::InitStarlaneRenderingBuffers
2015-05-11 14:52:21,546 DEBUG Client : MapWnd::InitStarlaneRenderingBuffers time: 5819
2015-05-11 14:52:21,553 DEBUG Client : ProductionWnd::UpdateQueue()
2015-05-11 14:52:24,995 DEBUG Client : HumanClientApp::HandleFocusChange()
--- snap ---
starlane rendering buffer on enqueuing something for production? looks like a mistake to me
does it recalculate the whole starlane network to refresh the estimated build time? that would kill usability for larger empires for sure O.o

[Geoff the Medio]

starlane rendering buffer on enqueuing something for production? looks like a mistake to me
does it recalculate the whole starlane network to refresh the estimated build time?

It recreates the rendering buffers for the starlanes, because when editing the queue, a group of systems may change from using all to not using all their available production output, and this is indicated on the map with differently coloured / sized starlanes.

[MatGB]

Ahh, that makes sense.

I'm mostly playing now on a better laptop than I was before, so I can play with larger setups before lag gets a problem, but definitely find production window is still a problem once my number of population centres gets too big, and given this machine is fairly well specced and "too big" is nowhere near the max available (350 systems on medium planets, 300 on high, but not counted how many planets I'd got), it would be good if something can be done to reduce the problem. I have recently given up on a couple games because the end-game lag was just frustrating, I had clearly "won" but hadn't completed.

[bigeagle]

i see why
so basically one would have to trigger that just at the beginning/end of a turn or manually, just turn it off or make it run in a background thread which would require much work i guess
or stop colonising many systems -> back to 'normal' scale :(
thanks, i will have a look at the code and try turning it off (and getting a working build environment) and recheck my savegame

one turn and ~ 20 colonies later it takes about 7 seconds already and any further expansion will be no fun

[marcOSX]

Well I had this problem on my Mac last year. Truth is my old core2duo MacBookPro was having trouble handling the production/science and design pages after 100 turns or so. I put more RAM, and people helped with the right click enqueing, but still it gets very slow later in the game.

Could it be possible to desactivate the starlane rendering while we are on the production, as the greyed out boxes clearly indicate the limit between things enqueud and in production. Maybe pressing the ? symbol would redraw the starlane map, as well as exiting from the production menu.

I also had the same problem with typing in design ships, with letters doubled, and such. For now, on the Mac version, it is OK.

[MatGB]

For the letter doubling issue specifically, in Options there's a set of form entries to decide how long it waits for a held key (or mouse button) before it decides you're holding it for repeat. Set these to zero and it never assumes you're holding for double.

[Dilvish]

I was about to go add some more logging to the Production queue code, when I it sunk in that even in this state it is still just taking 23 ms to do the queue projections (not the 23 seconds I initially read that as), so that is certainly not worth worrying about.

I tried running a multiplayer game to get up to where you were at here, and I must say, 5000 systems and 20 AIs make for some darn slow going. At turn 200 it is taking over a minute per turn-- just the post-combat processing takes about a minute for the server, and my client is reporting perhaps a minute or so being used per turn in total for a few meter update cycles. Starlane rendering is still just taking about 30ms per turn (and had started at ~10ms). So nothing like the 5.8 seconds that you are seeing. I have an i7, but it's just at 3.5 GHz compared to your i5's 4.2, so I think they are pretty similar performance-wise. The biggest AI still just had total PP of ~1k, and if we consider that is likely to be roughly proportional to the size of the empire (i.e., the number of starlanes affected by its supply-coloring), then even getting up to the size you are citing shouldn't take more than about 10 times this long, or around 300 ms, still nothing like 5800 ms. So there is something a little surprising there, and I suppose the production queue responsiveness is sensitive to this. As far as overall performance goes it seems substantially outweighed by the other per-turn processing required. How much are each of these turns taking to process (without entering any new orders, just hitting next-turn)?

In your other thread you characterized this as being about recalculating the Production Points, but I don't think that is quite really it-- it's the more broad issue that even simply putting something on the production queue could trigger other effects, and so all the effects need to be reprocessed when you change the queue, and that is probably what chews up the most time, not simply recalculating available Production Points. It is true that currently there are very few pieces of content that are dependent on whether or not you have something on the queue (all I can think of for now would be controlling whether or not you could add a new item to the queue, which is checked in a specific cycle, not requiring all the effects be rerun), so you might be able to code up some hack to substantially restrict what processing gets done there & still be ok for current content. I'm doubtful you would be able to find a streamlining that we could actually use for the mainline of the game, but perhaps you can figure out something for a special branch, "for_crazy_people_who_like_to_run_with_5000_systems"

[Scara]

Hi,
the here described Issue with the game freeze when adding stuff to the production queue sounds very familiar to me, although I experienced this slowdown already with much lower settings. As recommended I'm having a try with KDE desktop environment instead of XFCE and it's nice, but there is no difference in CPU load, when game is idle. In my last game with 200 systems I first noticed this production queue lag when adding stuff around Turn 200, that's ok for me. Well in my case it's a old Hardware issue I'm pretty sure now. 5000 Systems sounds like quite some task =) but I better not dare on my PC...

[bigeagle]

sadly the savegames are not compatible between the 0.4.4 windows relase and the current git master on linux
so i will have to play some hours before i can say something about the current status

is there some easy way to get the number of planets/colonies of an empire? the graphs don't show any numbers.

turn times of ~ 1 minute are ok in my opinion
space empires 5 easily takes more than 5 minutes on larger games once there are some battles and AI decisions to make (above all it crashes quite often and has other problems)
sins of a solar empires, being realtime makes it even worse, seems to have heavy slowdown even with medium sized games regardless of the hardware

if the client remains usable the turn times are not that important. the large the game, the longer it takes, that is inevitable. but you can continue to play and choose the scale you like.

edit: the game performance is much slower on current version and lubuntu 15.04
took 20 minutes to start the game, mainly universe generation
also the research screen with all techs is quite laggy with 11 fps
running on nearly the same hardware (just disks are different) and 'just' 3.8 ghz

[Dilvish]

Ah, I forgot to advise you to change the cmake build type from the default "Debug" instead to "Release"; that will make a big difference. I also just put in a change to one of the galaxy creation constraints that (especially for large galaxies) should help prevent a fair chunk of wasted time during galaxy creation.

[bigeagle]

i assume that is done by changing
set(FREEORION_RELEASE false)
to true in the CMakeLists.txt?

[Dilvish]

i assume that is done by changing
set(FREEORION_RELEASE false)
to true in the CMakeLists.txt?

I'm pretty sure that is just some kind of packaging flag. The key cmake variable you need to change is CMAKE_BUILD_TYPE; I usually do this via the curses cmake gui (ccmake) but on the command line you could just do

Code: Select all

cmake -DCMAKE_BUILD_TYPE=Release ..

[marcOSX]

Trying a huge map game on a Core 2 duo laptop, I can see what takes by far very long time.

1/Production queue as expected (likely because of the rendering of starlanes)
2/Change of focus of some planet (likely also for the update in starlane linked planets meters)
3/ship design (not clear why)
4/Encyclopedia calls
5/Planet suitability queries

[Scara]

Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz
8 GB RAM
Linux 3.13.0-37-generic
Freeorion build 2015-05-18.e5d1957

Hi folks,

having seen several players starting big maps I got curious and as well started a game with 2000 systems 15 AI and all on low, AI agression Manical.
I was rather suprised that it was running pretty good, comparable to my 200 System games! The first feelable slowdown first appeared around Turn 200 at the Productionqueuescreen.
From than on the lagtime was steadily increasing. I tried riddig colonies but it didn't help.
I still don't understand why the starlanerendering on a productionqueuechange would take so long.
The rendering is, to get it right the decision whether to color the starlane thick, thin or at all, its just the rendering not the computation?
If that is true I would really appreciate a button to turn starlanerendering off. I really use this only in early game to connect the supplysystem, at this stage using starlanerendering is important. Later in the game I maybe look at the starlanes but I completly ignore them being thick or thin. Only important thing is the color change on prodqueue empty, but there you also have a note at the top iconrow.
I included the part of the log concerning me adding a ship to the productionqueue, editing the number to be build and putting it to top position in queue. Takes about 8min, 6 of them are starlanerendering.

I stopped further colonization and invadeing around Turn 250 and produced only my best ships in large batches because the starfleetcontrol was working well even to the end.
Only event at starfleetcontrol I had to wait for, was splitting 10ship chunks from a 99Ship split stack.
Splitting the 99ships fleet is fast, seperating the first 10 Ships still is fast, seperating the following chuncks get more and more time consuming.
Splitting another 99Ships fleet in the same Turn results in a teetime. I guess thats the flyroute rendering for every of these ships, as you can't split a fleet in numberdefined chunks at one systems yet.
So I put Ships on some unused nodes in the AI player Empires to have a look what they where doing, secured my borders and let the game proceed on autoturn to see what happens.

I really appreciate the strenghening of the experimentors (Weakeningdevice, Troops and Shields) but for this game they only experimented on my patience still waiting for them to come out (Turn 620), sorry to come with them again
MatGB, you said there was a possible to change the Experiment Zero spawn chance, can you give me a clue were to find this setting? I would really like to experiment a bit with the Mad Scientists...
Interesting if it's possible to implement, a kind of dependency between the released monsters at the exp_outpost and the spawnrate of new ones, example the released monsters got killed pretty fast, feedback to the outpost to increase the productionrate, and the opposite maybe when all the spawned monsters still exist 20 rounds later. It looked like the experimentors have a static list of monsters they produce as soon as they come out, is that right?

I think the enormes increase of researchpoint needed for the last Research SoT is a little too big. In my current game I developed it at Turn 580. In default sized games you'll probably need much longer.

Question to Infrastructure, what is it for?
I read the pedia and it sounds very reasonable, but I'm still wondering what the number wants to say to me
Something that makes no sense to me is the fact that shipyard and orbital drydock give -10 each on the mostly overall structure of 20, resulting in 0. (?)

The sitrep scrollposition resets when switching to the Productionqueuescreen, to build something on a planet i got informed about by the sitrep.
Especially in large games I make heavy use of sitrep, but I always need to scroll down and look up the position I had been previously been going through the rep.
Is it possible to make sitrep remember its position while switching to the productionscreen?

In my actual game I have been using Sunshipfleets to penetrate deep into enemy AI space. I noticed my suns not loosings any fuel, tank remained on 7. Do the suns generate their fuel by their fusion processes? =)

[MatGB]

Infrastructure used to be "construction" and was linked to supply range, it was changed substantially and now it affects things like defence replenishment and similar (if you conquer two AI planets on the same turn, if one has drydock and the other no shipyards you'll notice the difference after 10 turns or so of shields and defence stats, depending on tech).

There have been discussions about what else to do with it, we know we want to do stuff with it, but we don't agree on what, at some point I might just do something based on my ideas.

Re Experimentors, in the file space_monster_spawn_fleets.txt in default you should find the following:

Code: Select all

MonsterFleet
    name = "SM_EXP_OUTPOST"
    ships = [
        "SM_EXP_OUTPOST"
    ]
    spawnrate = 1000.0
    spawnlimit = 1
    location = And [
        Contains And [
            Planet
            Not OwnedBy AnyEmpire
            Not Planet type = [Asteroids GasGiant]
        ]
        Not Contains Or [
                        HasSpecial name = "ANCIENT_RUINS_SPECIAL"
                        HasSpecial name = "PANOPTICON_SPECIAL"
                        HasSpecial name = "GAIA_SPECIAL"
                        ]
        Not WithinStarlaneJumps 6 And [
            Planet
            OwnedBy AnyEmpire
        ]
    ]

change the spawnlimit to whatever you want it to be, note that'll be the MAX, and each is limited as to where they can be, you could reduce the NotWithinStarlaneJumps as well if you want.

In Buildings.txt you can reduce their start time as well, on a large map it'll be massive so reducing it, or just killing the multiplier, is recommended.