GZStream and Platform incompatibility
Moderator: Committer
GZStream and Platform incompatibility
I had never noticed the GZStream library until Adrian just deleted the files, but that got me thinking-- the cross platform compatibility problem with boost archives is restricted to binary archives; the text archives are nicely portable, but folks didn't like their large size. If we just used the text archiving, but put it through GZStream before/after sending/receiving messages, or saving/loading files, it seems we could solve that problem. It would sure be nice to let Linux be game-compatible with Windows & MacOS...
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: GZStream and Platform incompatibility
I think there used to be a setting somewhere to enable compression of text archives, but I haven't been able to locate it or changes that removed it. There is this thread on this subject, though: viewtopic.php?f=9&t=4747
Re: GZStream and Platform incompatibility
That was very useful, thanks.Geoff the Medio wrote:I think there used to be a setting somewhere to enable compression of text archives, but I haven't been able to locate it or changes that removed it. There is this thread on this subject, though: viewtopic.php?f=9&t=4747
From that thread,
The conclusion in that thread was that "there is no point in compression if we can use binary serialization output," but it looks like the binary serialization compatibility problem hadn't been discovered yet.tzlaine wrote:We used to save and transmit our files compressed with zlib. ... We don't do that any more, because we want to be able to read the transmitted and save-file data for debugging. Eventually, we'll turn on binary serialization, and the size of transmitted and saved data will be <5% of what it is now. At that point, compression is probably a pessimization.
Now that we have discovered the platform compatibility problem with binary serialization, it seems to me that our debugging capabilities would be improved if in addition to the option for using text archives we also had an option to compress the text archives with zlib, as tzlaine indicated used to always be done. (And then those two options, for compressed text archives, could be the default, so that problem savegames that people post could be checked by any Dev.) I suppose that's why the QZStream code has been in our repository. If I can either dig up the old code for that, or come up with new code, is that something you'd agree would be good, Geoff?
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: GZStream and Platform incompatibility
Or it didn't exist yet. Versions of things change...Dilvish wrote:The conclusion in that thread was that "there is no point in compression if we can use binary serialization output," but it looks like the binary serialization compatibility problem hadn't been discovered yet.
Also, as often or more, people are using different boost versions to save and load, which can likely break serialization even on the same system. I'm not sure how affected text vs. binary is by this, though I'd guess binary would be more likely to be.
Not sure of your point here... Uncompressed archives would be a bit easier to open, though even compressed ones are probably easily openable using a standard archive reader program like 7-zip for debugging inspection purposes....it seems to me that our debugging capabilities would be improved if in addition to the option for using [uncompressed] text archives we also had an option to compress the text archives with zlib, as tzlaine indicated used to always be done.
The GZStream code predated the use of boost serialization. I don't think it would be the best solution now, as there are means to compress streams of text before within boost.I suppose that's why the QZStream code has been in our repository. If I can either dig up the old code for that, or come up with new code, is that something you'd agree would be good, Geoff?
http://stackoverflow.com/questions/1753 ... 81#5132081
http://www.boost.org/doc/libs/1_54_0/li ... /gzip.html
(and as linked in that other thread)
http://www.boost.org/doc/libs/1_43_0/li ... usage.html
It'd probably be fine to just set things up to always output and expected input as compressed text, for saving/loading, and possibly for interprocess messaging, although in the case of interprocess messaging, only selected parameters containing large blocks of data would likely be worth compressing.
Re: GZStream and Platform incompatibility
it's my understanding the current boost archives are backwards compatible quite a ways now; I think this is getting to be a fairly rare problem for us.Geoff the Medio wrote:Also, as often or more, people are using different boost versions to save and load, which can likely break serialization even on the same system. I'm not sure how affected text vs. binary is by this, though I'd guess binary would be more likely to be.
I mean that a lot of times folks post Windows savegames regarding some problem they've encountered, and I for one am not set up well on Windows to do anything with those. With a bit of effort I can get a system booted up in Windows, and update FO, and load the game to see what's going on, but even then I'm not set up in Windows to do any dev work like fixing an AI problem. Other times I see folks post savegames from Linux that you and other devs wouldn't be able to work with.Not sure of your point here... Uncompressed archives would be a bit easier to open, though even compressed ones are probably easily openable using a standard archive reader program like 7-zip for debugging inspection purposes....it seems to me that our debugging capabilities would be improved if in addition to the option for using [uncompressed] text archives we also had an option to compress the text archives with zlib, as tzlaine indicated used to always be done.
That looks to me like the same stuff as was available back at the time of the original thread you linked -- still requiring a special or locally compiled version of Boost, which was disliked: see boost gzip installation infoThe GZStream code predated the use of boost serialization. I don't think it would be the best solution now, as there are means to compress streams of text before within boost.I suppose that's why the QZStream code has been in our repository. If I can either dig up the old code for that, or come up with new code, is that something you'd agree would be good, Geoff?
http://stackoverflow.com/questions/1753 ... 81#5132081
http://www.boost.org/doc/libs/1_54_0/li ... /gzip.html
(and as linked in that other thread)
http://www.boost.org/doc/libs/1_43_0/li ... usage.html
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: GZStream and Platform incompatibility
But are previous boost archives forwards compatible? It's not just a matter of reading old saves when using two-way communication...Dilvish wrote:...the current boost archives are backwards compatible quite a ways now...
I'm not clear that it actually requires zlib to be available when building boost, or if it just adds a zlib dependency on the program being written using the gzip streams. FreeOrion already depends on zlib, so the latter would not be a problem. The gzip compressor filter code might be header-only, and just need to be linked against zlib.That looks to me like the same stuff as was available back at the time of the original thread you linked -- still requiring a special or locally compiled version of Boost, which was disliked...
Edit: Adding some instantiations of iostreams::gzip_compressor and iostreams::filtering_stream added link time dependency on the boost iostreams .lib and the boost zlib .lib. The latter at least suggests boost built with zlib would be needed.
So, i suppose the question is whether boost_zlib is currently a standard library available with Linux repository versions of boost? /Edit
- adrian_broher
- Programmer
- Posts: 1156
- Joined: Fri Mar 01, 2013 9:52 am
- Location: Germany
Re: GZStream and Platform incompatibility
This point is pretty much moot in my opinion. We are responsible for providing the dependencies for Mac and Windows anyway so we can decide how we configure them. For Linux we need to investigate what's the most common setup. How does boost upstream provide the libraries? Is boost iostreams gzip support enabled by default? If so it is fairly safe to assume that distributions provide boost iostreams with gzip support too.Dilvish wrote:still requiring a special or locally compiled version of Boost, which was disliked
- Fedora does enable the zlib dependency by default.
- RHEL and derivates always do the same as Fedora.
- Arch seems to enable the zlib dependency by default.
- Debian seems to enable the zlib dependency by default.
- Ubuntu does always the same as Debian.
- No idea about Suse, they don't provide package listings as it seems.
- Gentoo does enable the zlib dependency by default
Quote from a linked page in this link:
So it seems like the boost devs assume that zlib and libbz2 libaries are available by default on all Unix-ish systems. Which leaves Windows, where we have the control over dependencies anyway.http://www.boost.org/doc/libs/1_55_0/li ... .html#bjam
On most UNIX systems, it should not be necessary to set any of these variables, since the zlib and libbz2 headers and binaries are already installed in locations where they will be found automatically. On Windows, the zlib and bzip2 filters are disabled by default; to enable support for these filters, you will need to set variables indicating the location of the source files, if building from the source, or the location of the headers and binaries, if using pre-built binaries.
So my conclusion is: let us use the builtin boost iostreams gzip support if you want to compress the data.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz
- adrian_broher
- Programmer
- Posts: 1156
- Joined: Fri Mar 01, 2013 9:52 am
- Location: Germany
Re: GZStream and Platform incompatibility
Does this even matter? Are you intending to use an old client with a new server or the other way around?Geoff the Medio wrote:But are previous boost archives forwards compatible? It's not just a matter of reading old saves when using two-way communication...
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: GZStream and Platform incompatibility
A client built on Linux might use a different version of boost than the latest OSX or Windows build, even if they are based on the save SVN revision. If they are to interoperate, the archives need to be compatible not just between OSes, but in both directions between versions of boost.adrian_broher wrote:Does this even matter? Are you intending to use an old client with a new server or the other way around?Geoff the Medio wrote:But are previous boost archives forwards compatible? It's not just a matter of reading old saves when using two-way communication...
- adrian_broher
- Programmer
- Posts: 1156
- Joined: Fri Mar 01, 2013 9:52 am
- Location: Germany
Re: GZStream and Platform incompatibility
I see. But we (would) already have this problem because we already use different versions on different platforms.Geoff the Medio wrote:A client built on Linux might use a different version of boost than the latest OSX or Windows build, even if they are based on the save SVN revision. If they are to interoperate, the archives need to be compatible not just between OSes, but in both directions between versions of boost.
Regarding your question:
Back Versioning tells me that there is no guarantee for the 'wire format' for the archive to be forward compatible. The actual object serialization can add versions in a backward compatible way but that would require code changes on our side like adding version numbers to the classes.But are previous boost archives forwards compatible?
Last edited by adrian_broher on Sun Jan 12, 2014 5:27 pm, edited 1 time in total.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: GZStream and Platform incompatibility
Yes... So? The point is that switching to text archives might not fix the portability of saves in practice, due to the different boost versions being used.adrian_broher wrote:I see. But we (would) already have this problem because we already use different versions on different platforms.Geoff the Medio wrote:A client built on Linux might use a different version of boost than the latest OSX or Windows build, even if they are based on the save SVN revision. If they are to interoperate, the archives need to be compatible not just between OSes, but in both directions between versions of boost.
Changes to the data being serialized are not the concern. That will break serialization compatibility regardless of any other change, and adding version-awareness to the serialization code isn't worth the effort in the long-term. The issue is whether archives creating using different versions of boost, saving and restoring data from the same FreeOrion code, are interoperable....there is no guarantee for the 'wire format' for the archive to be backward compatible. The actual object serialization can add versions in a backward compatible way but that would require code changes on our side like adding version numbers to the classes.
Re: GZStream and Platform incompatibility
Please recall that we tested this several months ago, and yes, the use of text archives did solve the compatibility problems at least amongst the devs and with our prebuilt FO releases, which is pretty good coverage it seems.Geoff the Medio wrote:Yes... So? The point is that switching to text archives might not fix the portability of saves in practice, due to the different boost versions being used.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: GZStream and Platform incompatibility
I'm not saying it's not worth trying... Just don't expect it to fix every save and inter-OS multiplayer game compatibility problem.Dilvish wrote:Please recall...