Compressing Message data for transmission

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Post Reply
Message
Author
MindState
Space Krill
Posts: 6
Joined: Mon Jan 04, 2021 6:13 pm

Compressing Message data for transmission

#1 Post by MindState »

Earlier this year, I noticed that it takes a very long time for the FreeOrion server to transmit data clients during internet games. After further investigation, it seems the root causes are:
  • Limited upload speed of server's internet connection (in my case, about 1 Mbps)
  • Quantity of transmitted data
Not being able to do much about upload speed, I turned my attention to the quantity of data. The size of compressed saved game files suggested that the data was being transmitted uncompressed - and this seems to be the case. Working off of the release-v0.4.10 branch, I implemented compression for data transmission and ran a Ai-vs-AI test game over the LAN (500 systems, 20 AIs, 150 turns, running for 13 minutes)

If I didn't screw up the implementation and I interpreted my debug messages correctly, without compression the total data transmitted by the server would have been 7.2 GB with the largest message data size (toward the end of the run) being 13.2 MB. With compression this was reduced to 1.4 GB and 2.4 MB; uncompressed message sizes of 2.4 MB would have started showing about at around 2.5 minutes.

I have since rebased my quick-and-dirty branch off of master (see my msg_data_compression branch on GitHub.) EDIT: The test used the max compression option for zlib. I removed that from the current version; I don't know how to judge the trade-offs involved.

Is there any interest in having something like this merged into FreeOrion?

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Compressing Message data for transmission

#2 Post by Geoff the Medio »

In principle, yes, this could be good.

It needs to be backwards compatible, though, so older clients and older servers can talk to eachother.

It should also be optional, like the other XML compression settings, and perhaps default to only be used when sending data over a network and not between local clients and servers.

User avatar
Grummel7
Space Dragon
Posts: 335
Joined: Mon Oct 09, 2017 3:44 pm

Re: Compressing Message data for transmission

#3 Post by Grummel7 »

Perhaps using another port for compressed communication could solve the compatibility issue without changing the data format.

MindState
Space Krill
Posts: 6
Joined: Mon Jan 04, 2021 6:13 pm

Re: Compressing Message data for transmission

#4 Post by MindState »

I've tried my hand at implementing compressed data transmission through another port.

Revision (Branch)

Feedback welcome.

o01eg
Programmer
Posts: 2004
Joined: Sat Dec 10, 2011 5:46 am

Re: Compressing Message data for transmission

#5 Post by o01eg »

Next release is expected to be 0.5.* while latest stable 0.4.10.2. Maybe we could break compatibility?

Alternatively we could introduce new message types for big messages where content is compressed,
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-03-15.b3de094.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Compressing Message data for transmission

#6 Post by Vezzra »

o01eg wrote: Thu May 26, 2022 3:11 am Next release is expected to be 0.5.* while latest stable 0.4.10.2. Maybe we could break compatibility?
Not a concern. We're in alpha, breaking compatibility between releases (except bugfix/maintenance releases) is practically expected.

MindState
Space Krill
Posts: 6
Joined: Mon Jan 04, 2021 6:13 pm

Re: Compressing Message data for transmission

#7 Post by MindState »

My first attempt (linked in the first post) used a flag in the message header to indicate whether the data was compressed or not. I think it had the following changes:

* Message header is modified with an additional field indicating the compression algorithm used for the data; compression (or lack thereof) can be customized per message.
* During the game join request/reply, the client and server sends the other the compression algorithm that it prefers to receive data in; uncompressed is always accepted. This allows a client/server to receive a mix of uncompressed/compressed messages.
* The compression algorithm is user selectable (dropdown menu in options.)
* Message data below a certain size is always sent uncompressed, overriding algorithm setting; the threshold is user settable.
* Message data to localhost is always sent uncompressed, overriding algorithm setting

Simplification would be in order. I suspect it would be acceptable to have only one algorithm option (reducing handling to just "use compression yes/no", as in the network port variant) that did not change between major releases, and to hardcode the size threshold.

o01eg
Programmer
Posts: 2004
Joined: Sat Dec 10, 2011 5:46 am

Re: Compressing Message data for transmission

#8 Post by o01eg »

My approach of compression: https://github.com/freeorion/freeorion/pull/3870

It doesn't requires to allocate buffer for uncompressed data neither change message header.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-03-15.b3de094.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

MindState
Space Krill
Posts: 6
Joined: Mon Jan 04, 2021 6:13 pm

Re: Compressing Message data for transmission

#9 Post by MindState »

o01eg wrote: Thu Jun 02, 2022 6:26 pm My approach of compression: https://github.com/freeorion/freeorion/pull/3870

It doesn't requires to allocate buffer for uncompressed data neither change message header.
Oh, I like this.

An additional parameter could be added to each message creation function to toggle compression. I think zlib_compressor() has an option to set the compression level, one of which is for no compression. I'd need to try it, but my hope/expectation is that the output from a zlib_compressor() - whether it's compressed or not - can be fed right into a zlib_decompressor() without trouble.

After that, it'd just be a matter of finding and changing all of the message creation function calls.

o01eg
Programmer
Posts: 2004
Joined: Sat Dec 10, 2011 5:46 am

Re: Compressing Message data for transmission

#10 Post by o01eg »

I've added compression level parameter to GameStartMessage in my PR
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-03-15.b3de094.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Compressing Message data for transmission

#11 Post by Geoff the Medio »

o01eg wrote: Thu Jun 02, 2022 6:26 pm My approach of compression: https://github.com/freeorion/freeorion/pull/3870

It doesn't requires to allocate buffer for uncompressed data neither change message header.
In a related matter, I tried reworking the save game serialization with filtering streams for compression during serialization of the universe data. It was non-negligably slower than the current system: something like 1.5 s longer in a ~21 s long save process for a late game 14 MB (compressed) save file on my system. The current system pre-allocates a string buffer for the serialization text and then compresses that string into a second pre-allocated buffer. That inovles allocating a 537 MB (2^29 B) buffer to serialize into and a 67 MB (2^26 B) buffer for the compressed data. The streamed version can either do neither preallocated, or just preallocate the buffer for the compressed data. Memory use is (as expected) much less with the streamed version, but I'm not sure if that's worth the extra time saving.

Code: Select all

preallocated
14:39:51.013708 {0x0000607c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - collecting data      time:       12 ms
14:39:51.013708 {0x0000607c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - designs              time:       35 ms
14:39:51.013708 {0x0000607c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - vis / known          time:      716 ms
14:39:51.013708 {0x0000607c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - objects              time:     1099 ms
14:39:51.013708 {0x0000607c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - destroyed ids        time:       10 ms
14:39:51.013708 {0x0000607c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - latest known objects time:     6018 ms
14:39:51.013708 {0x0000607c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - stats                time:      530 ms
14:39:51.013708 {0x0000607c} [debug] timer : ScopedTimer.cpp:271 : Universe serializing                        time:     8423 ms


filter-streamed:
14:41:33.565159 {0x00002f0c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - collecting data      time:       12 ms
14:41:33.565159 {0x00002f0c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - designs              time:       42 ms
14:41:33.565159 {0x00002f0c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - vis / known          time:      822 ms
14:41:33.565159 {0x00002f0c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - objects              time:     1260 ms
14:41:33.565159 {0x00002f0c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - destroyed ids        time:       12 ms
14:41:33.565159 {0x00002f0c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - latest known objects time:     6956 ms
14:41:33.565159 {0x00002f0c} [debug] timer : ScopedTimer.cpp:261 : Universe serializing - stats                time:      628 ms
14:41:33.565159 {0x00002f0c} [debug] timer : ScopedTimer.cpp:271 : Universe serializing                        time:     9736 ms

o01eg
Programmer
Posts: 2004
Joined: Sat Dec 10, 2011 5:46 am

Re: Compressing Message data for transmission

#12 Post by o01eg »

Maybe reallocating buffer for bigger size cause slowness.
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-03-15.b3de094.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Compressing Message data for transmission

#13 Post by Geoff the Medio »

o01eg wrote: Sun Jun 05, 2022 2:49 pmMaybe reallocating buffer for bigger size cause slowness.
That's why the current implementation pre-allocates a big buffer to (de)serialize and (de)compress into.

o01eg
Programmer
Posts: 2004
Joined: Sat Dec 10, 2011 5:46 am

Re: Compressing Message data for transmission

#14 Post by o01eg »

Geoff the Medio wrote: Sun Jun 05, 2022 4:02 pm
o01eg wrote: Sun Jun 05, 2022 2:49 pmMaybe reallocating buffer for bigger size cause slowness.
That's why the current implementation pre-allocates a big buffer to (de)serialize and (de)compress into.
Maybe something like this will be helpful https://www.boost.org/doc/libs/1_69_0/d ... ambuf.html
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-03-15.b3de094.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

Post Reply