GZStream and Platform incompatibility

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderators: Committer, Committer

Post Reply
Message
Author
User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

GZStream and Platform incompatibility

#1 Post by Dilvish » Sat Jan 11, 2014 6:40 pm

I had never noticed the GZStream library until Adrian just deleted the files, but that got me thinking-- the cross platform compatibility problem with boost archives is restricted to binary archives; the text archives are nicely portable, but folks didn't like their large size. If we just used the text archiving, but put it through GZStream before/after sending/receiving messages, or saving/loading files, it seems we could solve that problem. It would sure be nice to let Linux be game-compatible with Windows & MacOS...
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12480
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GZStream and Platform incompatibility

#2 Post by Geoff the Medio » Sat Jan 11, 2014 6:53 pm

I think there used to be a setting somewhere to enable compression of text archives, but I haven't been able to locate it or changes that removed it. There is this thread on this subject, though: viewtopic.php?f=9&t=4747

User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: GZStream and Platform incompatibility

#3 Post by Dilvish » Sat Jan 11, 2014 10:51 pm

Geoff the Medio wrote:I think there used to be a setting somewhere to enable compression of text archives, but I haven't been able to locate it or changes that removed it. There is this thread on this subject, though: viewtopic.php?f=9&t=4747
That was very useful, thanks.

From that thread,
tzlaine wrote:We used to save and transmit our files compressed with zlib. ... We don't do that any more, because we want to be able to read the transmitted and save-file data for debugging. Eventually, we'll turn on binary serialization, and the size of transmitted and saved data will be <5% of what it is now. At that point, compression is probably a pessimization.
The conclusion in that thread was that "there is no point in compression if we can use binary serialization output," but it looks like the binary serialization compatibility problem hadn't been discovered yet.

Now that we have discovered the platform compatibility problem with binary serialization, it seems to me that our debugging capabilities would be improved if in addition to the option for using text archives we also had an option to compress the text archives with zlib, as tzlaine indicated used to always be done. (And then those two options, for compressed text archives, could be the default, so that problem savegames that people post could be checked by any Dev.) I suppose that's why the QZStream code has been in our repository. If I can either dig up the old code for that, or come up with new code, is that something you'd agree would be good, Geoff?
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12480
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GZStream and Platform incompatibility

#4 Post by Geoff the Medio » Sat Jan 11, 2014 11:27 pm

Dilvish wrote:The conclusion in that thread was that "there is no point in compression if we can use binary serialization output," but it looks like the binary serialization compatibility problem hadn't been discovered yet.
Or it didn't exist yet. Versions of things change...

Also, as often or more, people are using different boost versions to save and load, which can likely break serialization even on the same system. I'm not sure how affected text vs. binary is by this, though I'd guess binary would be more likely to be.
...it seems to me that our debugging capabilities would be improved if in addition to the option for using [uncompressed] text archives we also had an option to compress the text archives with zlib, as tzlaine indicated used to always be done.
Not sure of your point here... Uncompressed archives would be a bit easier to open, though even compressed ones are probably easily openable using a standard archive reader program like 7-zip for debugging inspection purposes.
I suppose that's why the QZStream code has been in our repository. If I can either dig up the old code for that, or come up with new code, is that something you'd agree would be good, Geoff?
The GZStream code predated the use of boost serialization. I don't think it would be the best solution now, as there are means to compress streams of text before within boost.

http://stackoverflow.com/questions/1753 ... 81#5132081
http://www.boost.org/doc/libs/1_54_0/li ... /gzip.html
(and as linked in that other thread)
http://www.boost.org/doc/libs/1_43_0/li ... usage.html

It'd probably be fine to just set things up to always output and expected input as compressed text, for saving/loading, and possibly for interprocess messaging, although in the case of interprocess messaging, only selected parameters containing large blocks of data would likely be worth compressing.

User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: GZStream and Platform incompatibility

#5 Post by Dilvish » Sun Jan 12, 2014 12:38 am

Geoff the Medio wrote:Also, as often or more, people are using different boost versions to save and load, which can likely break serialization even on the same system. I'm not sure how affected text vs. binary is by this, though I'd guess binary would be more likely to be.
it's my understanding the current boost archives are backwards compatible quite a ways now; I think this is getting to be a fairly rare problem for us.
...it seems to me that our debugging capabilities would be improved if in addition to the option for using [uncompressed] text archives we also had an option to compress the text archives with zlib, as tzlaine indicated used to always be done.
Not sure of your point here... Uncompressed archives would be a bit easier to open, though even compressed ones are probably easily openable using a standard archive reader program like 7-zip for debugging inspection purposes.
I mean that a lot of times folks post Windows savegames regarding some problem they've encountered, and I for one am not set up well on Windows to do anything with those. With a bit of effort I can get a system booted up in Windows, and update FO, and load the game to see what's going on, but even then I'm not set up in Windows to do any dev work like fixing an AI problem. Other times I see folks post savegames from Linux that you and other devs wouldn't be able to work with.
I suppose that's why the QZStream code has been in our repository. If I can either dig up the old code for that, or come up with new code, is that something you'd agree would be good, Geoff?
The GZStream code predated the use of boost serialization. I don't think it would be the best solution now, as there are means to compress streams of text before within boost.

http://stackoverflow.com/questions/1753 ... 81#5132081
http://www.boost.org/doc/libs/1_54_0/li ... /gzip.html
(and as linked in that other thread)
http://www.boost.org/doc/libs/1_43_0/li ... usage.html
That looks to me like the same stuff as was available back at the time of the original thread you linked -- still requiring a special or locally compiled version of Boost, which was disliked: see boost gzip installation info
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12480
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GZStream and Platform incompatibility

#6 Post by Geoff the Medio » Sun Jan 12, 2014 1:03 am

Dilvish wrote:...the current boost archives are backwards compatible quite a ways now...
But are previous boost archives forwards compatible? It's not just a matter of reading old saves when using two-way communication...
That looks to me like the same stuff as was available back at the time of the original thread you linked -- still requiring a special or locally compiled version of Boost, which was disliked...
I'm not clear that it actually requires zlib to be available when building boost, or if it just adds a zlib dependency on the program being written using the gzip streams. FreeOrion already depends on zlib, so the latter would not be a problem. The gzip compressor filter code might be header-only, and just need to be linked against zlib.

Edit: Adding some instantiations of iostreams::gzip_compressor and iostreams::filtering_stream added link time dependency on the boost iostreams .lib and the boost zlib .lib. The latter at least suggests boost built with zlib would be needed.

So, i suppose the question is whether boost_zlib is currently a standard library available with Linux repository versions of boost? /Edit

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: GZStream and Platform incompatibility

#7 Post by adrian_broher » Sun Jan 12, 2014 8:27 am

Dilvish wrote:still requiring a special or locally compiled version of Boost, which was disliked
This point is pretty much moot in my opinion. We are responsible for providing the dependencies for Mac and Windows anyway so we can decide how we configure them. For Linux we need to investigate what's the most common setup. How does boost upstream provide the libraries? Is boost iostreams gzip support enabled by default? If so it is fairly safe to assume that distributions provide boost iostreams with gzip support too.
  • Fedora does enable the zlib dependency by default.
  • RHEL and derivates always do the same as Fedora.
  • Arch seems to enable the zlib dependency by default.
  • Debian seems to enable the zlib dependency by default.
  • Ubuntu does always the same as Debian.
  • No idea about Suse, they don't provide package listings as it seems.
  • Gentoo does enable the zlib dependency by default
Quote from a linked page in this link:
http://www.boost.org/doc/libs/1_55_0/li ... .html#bjam

On most UNIX systems, it should not be necessary to set any of these variables, since the zlib and libbz2 headers and binaries are already installed in locations where they will be found automatically. On Windows, the zlib and bzip2 filters are disabled by default; to enable support for these filters, you will need to set variables indicating the location of the source files, if building from the source, or the location of the headers and binaries, if using pre-built binaries.
So it seems like the boost devs assume that zlib and libbz2 libaries are available by default on all Unix-ish systems. Which leaves Windows, where we have the control over dependencies anyway.

So my conclusion is: let us use the builtin boost iostreams gzip support if you want to compress the data.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: GZStream and Platform incompatibility

#8 Post by adrian_broher » Sun Jan 12, 2014 8:35 am

Geoff the Medio wrote:But are previous boost archives forwards compatible? It's not just a matter of reading old saves when using two-way communication...
Does this even matter? Are you intending to use an old client with a new server or the other way around?
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12480
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GZStream and Platform incompatibility

#9 Post by Geoff the Medio » Sun Jan 12, 2014 8:45 am

adrian_broher wrote:
Geoff the Medio wrote:But are previous boost archives forwards compatible? It's not just a matter of reading old saves when using two-way communication...
Does this even matter? Are you intending to use an old client with a new server or the other way around?
A client built on Linux might use a different version of boost than the latest OSX or Windows build, even if they are based on the save SVN revision. If they are to interoperate, the archives need to be compatible not just between OSes, but in both directions between versions of boost.

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: GZStream and Platform incompatibility

#10 Post by adrian_broher » Sun Jan 12, 2014 9:22 am

Geoff the Medio wrote:A client built on Linux might use a different version of boost than the latest OSX or Windows build, even if they are based on the save SVN revision. If they are to interoperate, the archives need to be compatible not just between OSes, but in both directions between versions of boost.
I see. But we (would) already have this problem because we already use different versions on different platforms.

Regarding your question:
But are previous boost archives forwards compatible?
Back Versioning tells me that there is no guarantee for the 'wire format' for the archive to be forward compatible. The actual object serialization can add versions in a backward compatible way but that would require code changes on our side like adding version numbers to the classes.
Last edited by adrian_broher on Sun Jan 12, 2014 5:27 pm, edited 1 time in total.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12480
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GZStream and Platform incompatibility

#11 Post by Geoff the Medio » Sun Jan 12, 2014 5:18 pm

adrian_broher wrote:
Geoff the Medio wrote:A client built on Linux might use a different version of boost than the latest OSX or Windows build, even if they are based on the save SVN revision. If they are to interoperate, the archives need to be compatible not just between OSes, but in both directions between versions of boost.
I see. But we (would) already have this problem because we already use different versions on different platforms.
Yes... So? The point is that switching to text archives might not fix the portability of saves in practice, due to the different boost versions being used.
...there is no guarantee for the 'wire format' for the archive to be backward compatible. The actual object serialization can add versions in a backward compatible way but that would require code changes on our side like adding version numbers to the classes.
Changes to the data being serialized are not the concern. That will break serialization compatibility regardless of any other change, and adding version-awareness to the serialization code isn't worth the effort in the long-term. The issue is whether archives creating using different versions of boost, saving and restoring data from the same FreeOrion code, are interoperable.

User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: GZStream and Platform incompatibility

#12 Post by Dilvish » Sun Jan 12, 2014 5:47 pm

Geoff the Medio wrote:Yes... So? The point is that switching to text archives might not fix the portability of saves in practice, due to the different boost versions being used.
Please recall that we tested this several months ago, and yes, the use of text archives did solve the compatibility problems at least amongst the devs and with our prebuilt FO releases, which is pretty good coverage it seems.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12480
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: GZStream and Platform incompatibility

#13 Post by Geoff the Medio » Sun Jan 12, 2014 6:01 pm

Dilvish wrote:Please recall...
I'm not saying it's not worth trying... Just don't expect it to fix every save and inter-OS multiplayer game compatibility problem.

Post Reply