Review save game Format
Posted: Sat Jan 05, 2019 6:00 pm
I would like to reopen the discussions on the save game format (https://www.freeorion.org/forum/viewtop ... lib#p66548) and here (https://www.freeorion.org/forum/viewtop ... f=9&t=4747) and possibly in other places as well.
As it stands, I believe the current xml+zlib+base64 compression method is suboptimal, for various reasons. Most importantly, it fails when it would be needed most: on large games.
In my case, I play with 400 to 800 star universes. With xml+zlib compression enabled the save game code (running on the server) will attempt to save xml+zlib. Due to quirks of the format, at first the xml serialization will have to be done to memory, to be included as base64 string in the "real"/wrapper savegame xml. At some point of the game, the memory serialization will start to fail every time. That's probably because it is not able to allocate a contiguous memory block of the required size. After this operation, which will typically take quite a few seconds, the code will resort to binary serialization, taking up another few seconds. There is no warning on this. When I started to compile my own version of FreeOrion, I was not ably to load these binary save games.
In my view, the whole xml+zlib+base64 approach is flawed, and should be replaced by the ideas discussed 8 years ago.
I would recommend saving the (original, uncompressed) xml format and the binary format in gzip compressed streams. For the xml format we could keep the current user configuration of saving uncompressed xml files.
This would have the following benefits:
- XML savegames would always work, regardless of configuration setting. There would be no need to silently and unexpectetly resort to binary serialization.
- There would be no out-of-memory condition on the server, leading to possibly other unexpected side effects, depending on what other threads are currently doing.
- Memory pressure on the server would be reduced, allowing to play larger games.
- There would be performance improvements, as we won't double-save binary saves.
- GZIP-compression would reduce xml files beyond what is currently possible ith xml+zlib+base64 (due to base64 blowing up the whole thing by 33.3%)
- GZIP-compression would reduce binary serialized file to 1/6 of their current size. E.g. instead of 100 MB per save game only 17 MB.
This would gave the following drawbacks:
- In one of the older discussions the question on external dependencies when using GZIP was raised. On my system (Windows) everything worked out of the box.
I have an implementation based on the current code available at https://github.com/OlafPettersson/freeo ... pSaveGames
On user configuration:
- There is an issue with user configuration. The Save Game-Options only take effect when FO is restarted. This is, I believe, due to the fact that the options are changed on the client, but will have to take effect on the server. I think this should I be either changed, so that the options take place immediately, or communicated clearly to the user.
As it stands, I believe the current xml+zlib+base64 compression method is suboptimal, for various reasons. Most importantly, it fails when it would be needed most: on large games.
In my case, I play with 400 to 800 star universes. With xml+zlib compression enabled the save game code (running on the server) will attempt to save xml+zlib. Due to quirks of the format, at first the xml serialization will have to be done to memory, to be included as base64 string in the "real"/wrapper savegame xml. At some point of the game, the memory serialization will start to fail every time. That's probably because it is not able to allocate a contiguous memory block of the required size. After this operation, which will typically take quite a few seconds, the code will resort to binary serialization, taking up another few seconds. There is no warning on this. When I started to compile my own version of FreeOrion, I was not ably to load these binary save games.
In my view, the whole xml+zlib+base64 approach is flawed, and should be replaced by the ideas discussed 8 years ago.
I would recommend saving the (original, uncompressed) xml format and the binary format in gzip compressed streams. For the xml format we could keep the current user configuration of saving uncompressed xml files.
This would have the following benefits:
- XML savegames would always work, regardless of configuration setting. There would be no need to silently and unexpectetly resort to binary serialization.
- There would be no out-of-memory condition on the server, leading to possibly other unexpected side effects, depending on what other threads are currently doing.
- Memory pressure on the server would be reduced, allowing to play larger games.
- There would be performance improvements, as we won't double-save binary saves.
- GZIP-compression would reduce xml files beyond what is currently possible ith xml+zlib+base64 (due to base64 blowing up the whole thing by 33.3%)
- GZIP-compression would reduce binary serialized file to 1/6 of their current size. E.g. instead of 100 MB per save game only 17 MB.
This would gave the following drawbacks:
- In one of the older discussions the question on external dependencies when using GZIP was raised. On my system (Windows) everything worked out of the box.
I have an implementation based on the current code available at https://github.com/OlafPettersson/freeo ... pSaveGames
On user configuration:
- There is an issue with user configuration. The Save Game-Options only take effect when FO is restarted. This is, I believe, due to the fact that the options are changed on the client, but will have to take effect on the server. I think this should I be either changed, so that the options take place immediately, or communicated clearly to the user.