Benchmarking

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Post Reply
Message
Author
o01eg
Programmer
Posts: 2052
Joined: Sat Dec 10, 2011 5:46 am

Benchmarking

#1 Post by o01eg »

I'm looking at #4943 and I think it could be useful to measure effectiveness of such changes.

I'm not sure about the best way to do it but google's https://github.com/google/benchmark looks nice and available on Linux distributives and could be added to Windows and MacOS SDK.

I've added checksum benchmark as an example but it doesn't measure anything significant https://github.com/o01eg/freeorion/tree/benchmarking
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-05-24.6c553c9.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13618
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Benchmarking

#2 Post by Geoff the Medio »

Microbenchmarking like (eg. with QuickBench) can be helpful, though for the content / effects system in FreeOrion, it's maybe not so useful, since the cases where we care about the performance are somewhat harder to test in isolation. For example, doing a full effects update on a big game a few hundred turns in, such as by changing a planet's focus in the UI. To test that, there needs to be a whole Universe and other gamestate set up (if not also a UI), and the only way to reproducibly and practically do that would be to have a save game that you load before starting the timing for each repetition of the benchmark. Just benchmarking small bits of the system in isolation might not give useful results, as it may not adequately replicate things like caching issues or the time it takes to look up objects to evaluate a condition, which could affect the results more in a "real world" test.

What I usually do to test performance effects of changes and to identify bottlenecks is run the game with the Visual Studio profiler attached, and cycle through systems in the UI by tapping the hotkey to do that, which causes an effects update to be run repeatedly. There are other profilers that can work similarly ("Tracy" seem popular recently).

Also, there are a bunch of lines in the logs about how long stuff takes, including some specific output of individual bits of content evalulation that took a long time, like:

Code: Select all

20:41:54.529167 {0x00008858} [debug] timer : ScopedTimer.cpp:67 : StoreTargetsAndCausesOfEffectsGroup < 347 >  cause type: ECT_SPECIES  specific cause: SP_FULVER  sources: 13  scope: And [    Planet    Not        VisibleToEmpire empire = Source.Owner    Not        Source    Not        OwnedBy empire = Source.Owner    WithinStarlaneJumps jumps = 2 condition =        Source] time:     9326 µs
20:41:54.555173 {0x0000235c} [debug] timer : ScopedTimer.cpp:67 : StoreTargetsAndCausesOfEffectsGroup < 981 >  cause type: ECT_SPECIES  specific cause: SP_FULVER  sources: 48  scope: And [    Planet    Not        VisibleToEmpire empire = Source.Owner    Not        Source    Not        OwnedBy empire = Source.Owner    WithinStarlaneJumps jumps = 2 condition =        Source] time:     19.4 ms
and

Code: Select all

20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - planet species  time:       24 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - ship species    time:        6 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - specials        time:        1 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - techs           time:        5 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - policies        time:        2 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - buildings       time:        4 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - ship hull/parts time:        4 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - fields          time:        0 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - eval waiting    time:        1 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:285 : Effect TargetSets Evaluation - reordering      time:        5 ms
20:41:54.566175 {0x0000ab04} [debug] timer : ScopedTimer.cpp:295 : Effect TargetSets Evaluation                   time:     59.2 ms
That said, microbenchmarking is fine with me if you want to set it up. I'm not sure how to do so for non-trivial cases for game mechanic internals like evaluating ValueRefs or Condition or executing effects, without having a populated Universe and other gamestate, though...

Specifically for CheckSum calculations, I don't really care if they are a bit slow, as they aren't run much or likely to substantially affect the player experience.

Also, to be clear, the point of the changes in the linked pull request is not just performance. It's also making the game mechanics classes testable at compile time, which also helps ensure they work correctly, not just quickly / less slowly.

Post Reply