boost and zlib

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderators: Committer, Committer

Message
Author
User avatar
tezeriusz
Space Squid
Posts: 75
Joined: Mon May 24, 2010 10:22 am
Location: Poland

boost and zlib

#1 Post by tezeriusz » Sun May 30, 2010 9:38 pm

Hello
I just wrote quick patch to allow compress saves. However i have question about boost included in game release under windows: Is BOOST_IOSTREAMS_ZLIB_HPP_INCLUDE defined? And if so (should be because FO is distributed with zlib as well) what to do with linux dynamic release? (standard boost under ubuntu is w/o zlib or any compressor AFAIK).

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12449
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: boost and zlib

#2 Post by Geoff the Medio » Sun May 30, 2010 11:26 pm

tezeriusz wrote:However i have question about boost included in game release under windows: Is BOOST_IOSTREAMS_ZLIB_HPP_INCLUDE defined?
The Windows release installer includes the pre-built boost libararies available from BoostPro. I don't know whether that define was set when the libraries were built.
And if so (should be because FO is distributed with zlib as well) what to do with linux dynamic release? (standard boost under ubuntu is w/o zlib or any compressor AFAIK).
What do you mean by "linux dynamic release" ?

User avatar
tezeriusz
Space Squid
Posts: 75
Joined: Mon May 24, 2010 10:22 am
Location: Poland

Re: boost and zlib

#3 Post by tezeriusz » Mon May 31, 2010 9:36 am

tezeriusz wrote:The Windows release installer includes the pre-built boost libararies available from BoostPro. I don't know whether that define was set when the libraries were built.
It is not defined in BoostPro release. Under windows you can check it by looking at filenames 'libboost_zlib-vc90-mt-gd-1_42.lib' was not found, being more accurate i didn't find any lib/dll with 'zlib' in it.
tezeriusz wrote:What do you mean by "linux dynamic release" ?
There is only static release my bad. So what should i do with that compress savegames. Should i assume that there will be prebuild boost with any filter or should I tray to write my own filter?(copy-past from boost?;]).

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12449
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: boost and zlib

#4 Post by Geoff the Medio » Mon May 31, 2010 4:49 pm

tezeriusz wrote:So what should i do with that compress savegames. Should i assume that there will be prebuild boost with any filter or should I tray to write my own filter?(copy-past from boost?;]).
I'm not familiar with the details of why you'd need to write a "filter", but to do file compression, since zlib is a dependency, so presumably it could be used. Alternatively, look at very old revisions of ServerApp.cpp and search the file for GZIP_SAVE_FILES_COMPRESSION_LEVEL to see how compression was done when this sourceforge feature request was written.

User avatar
tezeriusz
Space Squid
Posts: 75
Joined: Mon May 24, 2010 10:22 am
Location: Poland

Re: boost and zlib

#5 Post by tezeriusz » Mon May 31, 2010 6:21 pm

tezeriusz wrote:GZIP_SAVE_FILES_COMPRESSION_LEVEL to see how compression was done when this sourceforge feature request was written.
Great but it is using gzstream http://www.cs.unc.edu/Research/compgeom/gzstream/ with is not on soft_requirements wiki page http://www.freeorion.org/index.php/Comp ... quirements and boost and zlib is.
tezeriusz wrote:I'm not familiar with the details of why you'd need to write a "filter", but to do file compression, since zlib is a dependency, so presumably it could be used.
filters - http://www.boost.org/doc/libs/1_43_0/li ... usage.html
sure i can write nasty zlib code in C but in boost i can use "filter" and write compression in few lines:

boost::iostreams::filtering_streambuf<boost::iostreams::output> out;

out.push(boost::iostreams::gzip_compressor());
out.push(oFile);
boost::iostreams::copy(inFile, out);

So should i add new lib or do what to finish that patch? In my opinion if we have boost we should use it.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12449
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: boost and zlib

#6 Post by Geoff the Medio » Mon May 31, 2010 6:55 pm

tezeriusz wrote:...gzstream [...] is not on soft_requirements wiki page
That's because the gzstream code is already in the FreeOrion SVN repository, so isn't needed as a separate dependency. (This code actually uses zlib internally). The code hasn't actually been used for anything in about 3 years, and the code is slightly modified from code that's about 8 years old, based on the header comments, so should probably be replaced... But it's there, and doesn't need to be handled as a separate dependency.
So should i add new lib or do what to finish that patch? In my opinion if we have boost we should use it.
From what you've said, it seems that boost built with zlib support is non-standard. Particularly for Linux, I don't want to require a specialized version of Boost - with zlib support, if that's how things are actually set up - that will be different from what most people will get from their distro's repository. For Windows and OSX, we could start using Boost compiled with zlib support, but building Boost myself was causing a lot of problems, which is why I switched to using the BoostPro pre-built libraries, and I don't want to do this sort of thing differently between Linux and the other OSes.

What new library would you want to add, and why? Or was that covered in previous discussion?

Using Boost's filter framework is fine, if all the necessary pieces can be stitched together...

User avatar
tezeriusz
Space Squid
Posts: 75
Joined: Mon May 24, 2010 10:22 am
Location: Poland

Re: boost and zlib

#7 Post by tezeriusz » Mon May 31, 2010 8:02 pm

Geoff the Medio wrote:That's because the gzstream code is already in the FreeOrion SVN repository, so isn't needed as a separate dependency. (This code actually uses zlib internally). The code hasn't actually been used for anything in about 3 years, and the code is slightly modified from code that's about 8 years old, based on the header comments, so should probably be replaced... But it's there, and doesn't need to be handled as a separate dependency.
Lets drop GZStream for a moment.
tezeriusz wrote:From what you've said, it seems that boost built with zlib support is non-standard. Particularly for Linux, I don't want to require a specialized version of Boost - with zlib support, if that's how things are actually set up - that will be different from what most people will get from their distro's repository.
I just done a little investigation and result is:
there is zlib support in debian base systems (debian stable/unstable ubuntu(karmic/jaunty)) and in scientific linux
i didn't have it because i installed boost-1.42 from launchpad unofficial repo, now I'm using my own build 1.43. I apologize for misleading.
tezeriusz wrote:For Windows and OSX, we could start using Boost compiled with zlib support, but building Boost myself was causing a lot of problems, which is why I switched to using the BoostPro pre-built libraries, and I don't want to do this sort of thing differently between Linux and the other OSes.
There is BoostPro on linux?;]
We deliver static linked build so we can do whatever we want?
I just remember that i build boost 1.40/1.42 on winXP with mingw and that went smoothly. I will check on visual studio express 2008 tomorrow.
tezeriusz wrote:What new library would you want to add, and why? Or was that covered in previous discussion?
None now, I just didn't found util/GZStream.h btw how useful is that dir?

And I have another question because there is some lack in my boost/statechart knowledge. How i should deliver to user that there was something wrong with save/load process(deserialization/decompression/no more space on disk and so on)? How in that statemachine exceptions are handled?

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12449
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: boost and zlib

#8 Post by Geoff the Medio » Mon May 31, 2010 9:01 pm

tezeriusz wrote:I just done a little investigation and result is:
there is zlib support in debian base systems (debian stable/unstable ubuntu(karmic/jaunty)) and in scientific linux
i didn't have it because i installed boost-1.42 from launchpad unofficial repo, now I'm using my own build 1.43. I apologize for misleading.
That's encouraging, though I'm still reluctant to complicate the dependencies even more by requiring a particular variant of Boost that not everyone will have...
There is BoostPro on linux?;]
Not that I know of... (Not sure why you're asking...)
We deliver static linked build so we can do whatever we want?
That's fine for the static build, but various people like to build FreeOrion for themselves, and there are already numerous issues with dependency version requirements, including versions of Boost different from what people have in their repositories and that are mutually incompatible. I don't want to add another specific requirement (Boost with zlib support) if it's not (nearly) standard to build it that way.
I just remember that i build boost 1.40/1.42 on winXP with mingw and that went smoothly. I will check on visual studio express 2008 tomorrow.
It was an issue with bjam on my particular system. I think it probably does work for most people, but there was some specific issue with my setup that was causing bjam to fail, and after spending many hours on it and copious googling, there was no apparent fix available, so I switched to using the the BoostPro builds.

It might be worthing checking whether BoostPro actually does ship the zlib-enabled versions of the relevant libraries as well; I may have just not installed and distributed those variants.
tezeriusz wrote:What new library would you want to add, and why? Or was that covered in previous discussion?
None now, I just didn't found util/GZStream.h btw how useful is that dir?
I don't understand your question. GZStream.h isn't a "dir"...
And I have another question because there is some lack in my boost/statechart knowledge. How i should deliver to user that there was something wrong with save/load process(deserialization/decompression/no more space on disk and so on)? How in that statemachine exceptions are handled?
I don't think is really an issue about the state machine...

Both auto saves and manual saves are initiated by the human client by calling HumanClientApp::SaveGame. This sends a message to the server to initiate a save.

On the server, in WaitingForSaveData::react, when the last needed save data response in received from clients, the SaveGame function is called, which writes out a file. Whether or not that works, the same state machine transition should happen: back to WaitingForTurnEndIdle.

If something goes wrong in SaveGame, you could abort the function and send a message to human clients to say that the save failed. This doesn't require any state machine special cases, though...

User avatar
tezeriusz
Space Squid
Posts: 75
Joined: Mon May 24, 2010 10:22 am
Location: Poland

Re: boost and zlib

#9 Post by tezeriusz » Tue Jun 01, 2010 11:02 am

Geoff the Medio wrote:Quote:
There is BoostPro on linux?;]

Not that I know of... (Not sure why you're asking...)
Just wanted to point - if you want unified solution you need to provide it yourself.
Geoff the Medio wrote:That's fine for the static build, but various people like to build FreeOrion for themselves, and there are already numerous issues with dependency version requirements, including versions of Boost different from what people have in their repositories and that are mutually incompatible
Did I saw FreeOrion SDK's on win32(?) and macOSX? So there should be 'tweaked' boost in it.
Geoff the Medio wrote: It might be worthing checking whether BoostPro actually does ship the zlib-enabled versions of the relevant libraries as well; I may have just not installed and distributed those variants.
I didn't find zlib build from BoostPro. However i build it myself on visual studio express 2008.
Solution 1:
zlib 1.2.3 in d:/zlib dir and then from "visual studio command prompt" in boost dir:
".\bjam.exe --prefix="d:\boost-test" -sZLIB_INCLUDE="d:/zlib" -sZLIB_SOURCE="d:/zlib" --with-iostreams --build-type=complete stage install"

Solution 2:
zlib 1.2.5 in d:/zlib5 dir and then from "visual studio command prompt":
go to d:\boost_1_43_0\libs\iostreams\build\Jamfile.v2 and remove 'gzio' from 'local z =' array;
sed -e 's/gzio//g' will do the trick.
".\bjam.exe --prefix="d:\boost-test" -sZLIB_INCLUDE="d:/zlib5" -sZLIB_SOURCE="d:/zlib5" --with-iostreams --build-type=complete stage install"

and thats all folks - we have mega customized release of boost ;]
more info about customized build on windows here-> http://www.pion.org/files/pion-platform ... EADME.msvc

Do we have any translations in FO? If yes do they require boost/ICU iconv gettext?
Geoff the Medio wrote:Quote:
tezeriusz wrote:
What new library would you want to add, and why? Or was that covered in previous discussion?

None now, I just didn't found util/GZStream.h btw how useful is that dir?

I don't understand your question. GZStream.h isn't a "dir"...
Sure but 'util' is and that was question about freshness.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12449
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: boost and zlib

#10 Post by Geoff the Medio » Tue Jun 01, 2010 5:18 pm

tezeriusz wrote:Just wanted to point - if you want unified solution you need to provide it yourself.
I would be more comfortable putting some code into the FreeOrion SVN tree than assuming a dependency is built a particular way.
Geoff the Medio wrote:That's fine for the static build, but various people like to build FreeOrion for themselves, and there are already numerous issues with dependency version requirements, including versions of Boost different from what people have in their repositories and that are mutually incompatible
Did I saw FreeOrion SDK's on win32(?) and macOSX? So there should be 'tweaked' boost in it.
Yes, there are SDKs for Win32 and OSX, but the problem I'm worried about is on Linux, where anyone building FreeOrion will likely get Boost from their repository, and won't want, or won't be able, to have multiple versions with and without zlib support.
I didn't find zlib build from BoostPro. However i build it myself on visual studio express 2008.
[...]
and thats all folks - we have mega customized release of boost ;]
I didn't say I didn't know how to build Boost; I said it just didn't work on my machine. That was a few versions ago, and it might work now, though... But it's moot if there are still the Linux issues.
Do we have any translations in FO? If yes do they require boost/ICU iconv gettext?
There are translations in the various ??_stringtable.txt files in /default. You can switch the language in the options screen. Gettext is not used.
I don't understand your question. GZStream.h isn't a "dir"...
Sure but 'util' is and that was question about freshness.
I still don't know what the question was...

User avatar
tezeriusz
Space Squid
Posts: 75
Joined: Mon May 24, 2010 10:22 am
Location: Poland

Re: boost and zlib

#11 Post by tezeriusz » Tue Jun 01, 2010 7:52 pm

Geoff the Medio wrote:Yes, there are SDKs for Win32 and OSX, but the problem I'm worried about is on Linux, where anyone building FreeOrion will likely get Boost from their repository, and won't want, or won't be able, to have multiple versions with and without zlib support.
Wait a sec, ubuntu/debian have it, arch/gentoo is no problem they can recompile anytime - what Linux users are you referring too? Somone from FO community. Eh so much resistance i will use that zlib include check from boost and tray not to break compatibility with previous version.
Geoff the Medio wrote:I didn't say I didn't know how to build Boost; I said it just didn't work on my machine. That was a few versions ago, and it might work now, though... But it's moot if there are still the Linux issues.
I didn't know if it work on my machine too. And in my opinion there is not any linux boost-zlib issue.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12449
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: boost and zlib

#12 Post by Geoff the Medio » Tue Jun 01, 2010 8:50 pm

tezeriusz wrote:Wait a sec, ubuntu/debian have it, arch/gentoo is no problem they can recompile anytime - what Linux users are you referring too?
I don't have any specific distro in mind, and I don't know the various distros well enough to know which would be a problem, but I'm just a bit paranoid about doing something non-standard due to all the problems we've had due to Boost version issues and the inability of some people to use different versions than what comes from their distro's repository. Maybe it's not an issue at all on Linux, and every distro's repository has boost with zlib support... but I'd like to know if that's the case.
Somone from FO community. Eh so much resistance i will use that zlib include check from boost and tray not to break compatibility with previous version.
If you can make things fall back to no-compression when the boost zlip define isn't set, that would be nice. Will the zlib-enabled build be able to read and write uncompressed saves in this case?
And in my opinion there is not any linux boost-zlib issue.
OK. Hopefully that's accurate.

User avatar
tzlaine
Programming Lead Emeritus
Posts: 1092
Joined: Thu Jun 26, 2003 1:33 pm

Re: boost and zlib

#13 Post by tzlaine » Wed Jun 02, 2010 9:58 pm

We used to save and transmit our files compressed with zlib. Notice the "#include <zlib.h>" cruft near the top of Message.cpp.

We don't do that any more, because we want to be able to read the transmitted and save-file data for debugging.

Eventually, we'll turn on binary serialization, and the size of transmitted and saved data will be <5% of what it is now. At that point, compression is probably a pessimization.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12449
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: boost and zlib

#14 Post by Geoff the Medio » Thu Jun 03, 2010 12:34 am

tzlaine wrote:We used to save and transmit our files compressed with zlib. Notice the "#include <zlib.h>" cruft near the top of Message.cpp.

We don't do that any more, because we want to be able to read the transmitted and save-file data for debugging.
Would zlib-compressed save files be difficult to open with a standard zip file manager?
Eventually, we'll turn on binary serialization, and the size of transmitted and saved data will be <5% of what it is now. At that point, compression is probably a pessimization.
We should probably test that before assuming save compression won't be beneficial.

I suspect that the time to compress and decompress turn updates would be more than any benefit from smaller amounts of data being sent (though that should also probably be tested...)

User avatar
tezeriusz
Space Squid
Posts: 75
Joined: Mon May 24, 2010 10:22 am
Location: Poland

Re: boost and zlib

#15 Post by tezeriusz » Tue Jun 08, 2010 4:16 pm

Geoff the Medio wrote:Would zlib-compressed save files be difficult to open with a standard zip file manager?
I can decompress with standard tool, save file compressed with:
zip - no
gzip - yes
bz2 - yes
Geoff the Medio wrote:We should probably test that before assuming save compression won't be beneficial.
Test results:
UNCOMP XML with streambuf - iterations:10000
time0 = 9785
filesize0 = 3674
ZLIB XML with streambuf - iterations:10000
time1 = 14568
filesize1 = 727
GZIP XML with streambuf - iterations:10000
time2 = 15818
filesize2 = 739
BZ2 XML with streambuf - iterations:10000
time3 = 23844
filesize3 = 804
UNCOMP XML w/o streambuf - iterations:10000
time5 = 8546
filesize5 = 3698
UNCOMP bin w/o streambuf - iterations:10000
time6 = 4771
filesize6 = 510

boost is fun ;]
Attachments
tests.tar.gz
test to check various compress algorithms time and file size.
(4.83 KiB) Downloaded 65 times

Post Reply