boost and zlib
Moderator: Committer
boost and zlib
Hello
I just wrote quick patch to allow compress saves. However i have question about boost included in game release under windows: Is BOOST_IOSTREAMS_ZLIB_HPP_INCLUDE defined? And if so (should be because FO is distributed with zlib as well) what to do with linux dynamic release? (standard boost under ubuntu is w/o zlib or any compressor AFAIK).
I just wrote quick patch to allow compress saves. However i have question about boost included in game release under windows: Is BOOST_IOSTREAMS_ZLIB_HPP_INCLUDE defined? And if so (should be because FO is distributed with zlib as well) what to do with linux dynamic release? (standard boost under ubuntu is w/o zlib or any compressor AFAIK).
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: boost and zlib
The Windows release installer includes the pre-built boost libararies available from BoostPro. I don't know whether that define was set when the libraries were built.tezeriusz wrote:However i have question about boost included in game release under windows: Is BOOST_IOSTREAMS_ZLIB_HPP_INCLUDE defined?
What do you mean by "linux dynamic release" ?And if so (should be because FO is distributed with zlib as well) what to do with linux dynamic release? (standard boost under ubuntu is w/o zlib or any compressor AFAIK).
Re: boost and zlib
It is not defined in BoostPro release. Under windows you can check it by looking at filenames 'libboost_zlib-vc90-mt-gd-1_42.lib' was not found, being more accurate i didn't find any lib/dll with 'zlib' in it.tezeriusz wrote:The Windows release installer includes the pre-built boost libararies available from BoostPro. I don't know whether that define was set when the libraries were built.
There is only static release my bad. So what should i do with that compress savegames. Should i assume that there will be prebuild boost with any filter or should I tray to write my own filter?(copy-past from boost?;]).tezeriusz wrote:What do you mean by "linux dynamic release" ?
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: boost and zlib
I'm not familiar with the details of why you'd need to write a "filter", but to do file compression, since zlib is a dependency, so presumably it could be used. Alternatively, look at very old revisions of ServerApp.cpp and search the file for GZIP_SAVE_FILES_COMPRESSION_LEVEL to see how compression was done when this sourceforge feature request was written.tezeriusz wrote:So what should i do with that compress savegames. Should i assume that there will be prebuild boost with any filter or should I tray to write my own filter?(copy-past from boost?;]).
Re: boost and zlib
Great but it is using gzstream http://www.cs.unc.edu/Research/compgeom/gzstream/ with is not on soft_requirements wiki page http://www.freeorion.org/index.php/Comp ... quirements and boost and zlib is.tezeriusz wrote:GZIP_SAVE_FILES_COMPRESSION_LEVEL to see how compression was done when this sourceforge feature request was written.
filters - http://www.boost.org/doc/libs/1_43_0/li ... usage.htmltezeriusz wrote:I'm not familiar with the details of why you'd need to write a "filter", but to do file compression, since zlib is a dependency, so presumably it could be used.
sure i can write nasty zlib code in C but in boost i can use "filter" and write compression in few lines:
boost::iostreams::filtering_streambuf<boost::iostreams::output> out;
out.push(boost::iostreams::gzip_compressor());
out.push(oFile);
boost::iostreams::copy(inFile, out);
So should i add new lib or do what to finish that patch? In my opinion if we have boost we should use it.
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: boost and zlib
That's because the gzstream code is already in the FreeOrion SVN repository, so isn't needed as a separate dependency. (This code actually uses zlib internally). The code hasn't actually been used for anything in about 3 years, and the code is slightly modified from code that's about 8 years old, based on the header comments, so should probably be replaced... But it's there, and doesn't need to be handled as a separate dependency.tezeriusz wrote:...gzstream [...] is not on soft_requirements wiki page
From what you've said, it seems that boost built with zlib support is non-standard. Particularly for Linux, I don't want to require a specialized version of Boost - with zlib support, if that's how things are actually set up - that will be different from what most people will get from their distro's repository. For Windows and OSX, we could start using Boost compiled with zlib support, but building Boost myself was causing a lot of problems, which is why I switched to using the BoostPro pre-built libraries, and I don't want to do this sort of thing differently between Linux and the other OSes.So should i add new lib or do what to finish that patch? In my opinion if we have boost we should use it.
What new library would you want to add, and why? Or was that covered in previous discussion?
Using Boost's filter framework is fine, if all the necessary pieces can be stitched together...
Re: boost and zlib
Lets drop GZStream for a moment.Geoff the Medio wrote:That's because the gzstream code is already in the FreeOrion SVN repository, so isn't needed as a separate dependency. (This code actually uses zlib internally). The code hasn't actually been used for anything in about 3 years, and the code is slightly modified from code that's about 8 years old, based on the header comments, so should probably be replaced... But it's there, and doesn't need to be handled as a separate dependency.
I just done a little investigation and result is:tezeriusz wrote:From what you've said, it seems that boost built with zlib support is non-standard. Particularly for Linux, I don't want to require a specialized version of Boost - with zlib support, if that's how things are actually set up - that will be different from what most people will get from their distro's repository.
there is zlib support in debian base systems (debian stable/unstable ubuntu(karmic/jaunty)) and in scientific linux
i didn't have it because i installed boost-1.42 from launchpad unofficial repo, now I'm using my own build 1.43. I apologize for misleading.
There is BoostPro on linux?;]tezeriusz wrote:For Windows and OSX, we could start using Boost compiled with zlib support, but building Boost myself was causing a lot of problems, which is why I switched to using the BoostPro pre-built libraries, and I don't want to do this sort of thing differently between Linux and the other OSes.
We deliver static linked build so we can do whatever we want?
I just remember that i build boost 1.40/1.42 on winXP with mingw and that went smoothly. I will check on visual studio express 2008 tomorrow.
None now, I just didn't found util/GZStream.h btw how useful is that dir?tezeriusz wrote:What new library would you want to add, and why? Or was that covered in previous discussion?
And I have another question because there is some lack in my boost/statechart knowledge. How i should deliver to user that there was something wrong with save/load process(deserialization/decompression/no more space on disk and so on)? How in that statemachine exceptions are handled?
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: boost and zlib
That's encouraging, though I'm still reluctant to complicate the dependencies even more by requiring a particular variant of Boost that not everyone will have...tezeriusz wrote:I just done a little investigation and result is:
there is zlib support in debian base systems (debian stable/unstable ubuntu(karmic/jaunty)) and in scientific linux
i didn't have it because i installed boost-1.42 from launchpad unofficial repo, now I'm using my own build 1.43. I apologize for misleading.
Not that I know of... (Not sure why you're asking...)There is BoostPro on linux?;]
That's fine for the static build, but various people like to build FreeOrion for themselves, and there are already numerous issues with dependency version requirements, including versions of Boost different from what people have in their repositories and that are mutually incompatible. I don't want to add another specific requirement (Boost with zlib support) if it's not (nearly) standard to build it that way.We deliver static linked build so we can do whatever we want?
It was an issue with bjam on my particular system. I think it probably does work for most people, but there was some specific issue with my setup that was causing bjam to fail, and after spending many hours on it and copious googling, there was no apparent fix available, so I switched to using the the BoostPro builds.I just remember that i build boost 1.40/1.42 on winXP with mingw and that went smoothly. I will check on visual studio express 2008 tomorrow.
It might be worthing checking whether BoostPro actually does ship the zlib-enabled versions of the relevant libraries as well; I may have just not installed and distributed those variants.
I don't understand your question. GZStream.h isn't a "dir"...None now, I just didn't found util/GZStream.h btw how useful is that dir?tezeriusz wrote:What new library would you want to add, and why? Or was that covered in previous discussion?
I don't think is really an issue about the state machine...And I have another question because there is some lack in my boost/statechart knowledge. How i should deliver to user that there was something wrong with save/load process(deserialization/decompression/no more space on disk and so on)? How in that statemachine exceptions are handled?
Both auto saves and manual saves are initiated by the human client by calling HumanClientApp::SaveGame. This sends a message to the server to initiate a save.
On the server, in WaitingForSaveData::react, when the last needed save data response in received from clients, the SaveGame function is called, which writes out a file. Whether or not that works, the same state machine transition should happen: back to WaitingForTurnEndIdle.
If something goes wrong in SaveGame, you could abort the function and send a message to human clients to say that the save failed. This doesn't require any state machine special cases, though...
Re: boost and zlib
Just wanted to point - if you want unified solution you need to provide it yourself.Geoff the Medio wrote:Quote:
There is BoostPro on linux?;]
Not that I know of... (Not sure why you're asking...)
Did I saw FreeOrion SDK's on win32(?) and macOSX? So there should be 'tweaked' boost in it.Geoff the Medio wrote:That's fine for the static build, but various people like to build FreeOrion for themselves, and there are already numerous issues with dependency version requirements, including versions of Boost different from what people have in their repositories and that are mutually incompatible
I didn't find zlib build from BoostPro. However i build it myself on visual studio express 2008.Geoff the Medio wrote: It might be worthing checking whether BoostPro actually does ship the zlib-enabled versions of the relevant libraries as well; I may have just not installed and distributed those variants.
Solution 1:
zlib 1.2.3 in d:/zlib dir and then from "visual studio command prompt" in boost dir:
".\bjam.exe --prefix="d:\boost-test" -sZLIB_INCLUDE="d:/zlib" -sZLIB_SOURCE="d:/zlib" --with-iostreams --build-type=complete stage install"
Solution 2:
zlib 1.2.5 in d:/zlib5 dir and then from "visual studio command prompt":
go to d:\boost_1_43_0\libs\iostreams\build\Jamfile.v2 and remove 'gzio' from 'local z =' array;
sed -e 's/gzio//g' will do the trick.
".\bjam.exe --prefix="d:\boost-test" -sZLIB_INCLUDE="d:/zlib5" -sZLIB_SOURCE="d:/zlib5" --with-iostreams --build-type=complete stage install"
and thats all folks - we have mega customized release of boost ;]
more info about customized build on windows here-> http://www.pion.org/files/pion-platform ... EADME.msvc
Do we have any translations in FO? If yes do they require boost/ICU iconv gettext?
Sure but 'util' is and that was question about freshness.Geoff the Medio wrote:Quote:
tezeriusz wrote:
What new library would you want to add, and why? Or was that covered in previous discussion?
None now, I just didn't found util/GZStream.h btw how useful is that dir?
I don't understand your question. GZStream.h isn't a "dir"...
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: boost and zlib
I would be more comfortable putting some code into the FreeOrion SVN tree than assuming a dependency is built a particular way.tezeriusz wrote:Just wanted to point - if you want unified solution you need to provide it yourself.
Yes, there are SDKs for Win32 and OSX, but the problem I'm worried about is on Linux, where anyone building FreeOrion will likely get Boost from their repository, and won't want, or won't be able, to have multiple versions with and without zlib support.Did I saw FreeOrion SDK's on win32(?) and macOSX? So there should be 'tweaked' boost in it.Geoff the Medio wrote:That's fine for the static build, but various people like to build FreeOrion for themselves, and there are already numerous issues with dependency version requirements, including versions of Boost different from what people have in their repositories and that are mutually incompatible
I didn't say I didn't know how to build Boost; I said it just didn't work on my machine. That was a few versions ago, and it might work now, though... But it's moot if there are still the Linux issues.I didn't find zlib build from BoostPro. However i build it myself on visual studio express 2008.
[...]
and thats all folks - we have mega customized release of boost ;]
There are translations in the various ??_stringtable.txt files in /default. You can switch the language in the options screen. Gettext is not used.Do we have any translations in FO? If yes do they require boost/ICU iconv gettext?
I still don't know what the question was...Sure but 'util' is and that was question about freshness.I don't understand your question. GZStream.h isn't a "dir"...
Re: boost and zlib
Wait a sec, ubuntu/debian have it, arch/gentoo is no problem they can recompile anytime - what Linux users are you referring too? Somone from FO community. Eh so much resistance i will use that zlib include check from boost and tray not to break compatibility with previous version.Geoff the Medio wrote:Yes, there are SDKs for Win32 and OSX, but the problem I'm worried about is on Linux, where anyone building FreeOrion will likely get Boost from their repository, and won't want, or won't be able, to have multiple versions with and without zlib support.
I didn't know if it work on my machine too. And in my opinion there is not any linux boost-zlib issue.Geoff the Medio wrote:I didn't say I didn't know how to build Boost; I said it just didn't work on my machine. That was a few versions ago, and it might work now, though... But it's moot if there are still the Linux issues.
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: boost and zlib
I don't have any specific distro in mind, and I don't know the various distros well enough to know which would be a problem, but I'm just a bit paranoid about doing something non-standard due to all the problems we've had due to Boost version issues and the inability of some people to use different versions than what comes from their distro's repository. Maybe it's not an issue at all on Linux, and every distro's repository has boost with zlib support... but I'd like to know if that's the case.tezeriusz wrote:Wait a sec, ubuntu/debian have it, arch/gentoo is no problem they can recompile anytime - what Linux users are you referring too?
If you can make things fall back to no-compression when the boost zlip define isn't set, that would be nice. Will the zlib-enabled build be able to read and write uncompressed saves in this case?Somone from FO community. Eh so much resistance i will use that zlib include check from boost and tray not to break compatibility with previous version.
OK. Hopefully that's accurate.And in my opinion there is not any linux boost-zlib issue.
Re: boost and zlib
We used to save and transmit our files compressed with zlib. Notice the "#include <zlib.h>" cruft near the top of Message.cpp.
We don't do that any more, because we want to be able to read the transmitted and save-file data for debugging.
Eventually, we'll turn on binary serialization, and the size of transmitted and saved data will be <5% of what it is now. At that point, compression is probably a pessimization.
We don't do that any more, because we want to be able to read the transmitted and save-file data for debugging.
Eventually, we'll turn on binary serialization, and the size of transmitted and saved data will be <5% of what it is now. At that point, compression is probably a pessimization.
- Geoff the Medio
- Programming, Design, Admin
- Posts: 13587
- Joined: Wed Oct 08, 2003 1:33 am
- Location: Munich
Re: boost and zlib
Would zlib-compressed save files be difficult to open with a standard zip file manager?tzlaine wrote:We used to save and transmit our files compressed with zlib. Notice the "#include <zlib.h>" cruft near the top of Message.cpp.
We don't do that any more, because we want to be able to read the transmitted and save-file data for debugging.
We should probably test that before assuming save compression won't be beneficial.Eventually, we'll turn on binary serialization, and the size of transmitted and saved data will be <5% of what it is now. At that point, compression is probably a pessimization.
I suspect that the time to compress and decompress turn updates would be more than any benefit from smaller amounts of data being sent (though that should also probably be tested...)
Re: boost and zlib
I can decompress with standard tool, save file compressed with:Geoff the Medio wrote:Would zlib-compressed save files be difficult to open with a standard zip file manager?
zip - no
gzip - yes
bz2 - yes
Test results:Geoff the Medio wrote:We should probably test that before assuming save compression won't be beneficial.
UNCOMP XML with streambuf - iterations:10000
time0 = 9785
filesize0 = 3674
ZLIB XML with streambuf - iterations:10000
time1 = 14568
filesize1 = 727
GZIP XML with streambuf - iterations:10000
time2 = 15818
filesize2 = 739
BZ2 XML with streambuf - iterations:10000
time3 = 23844
filesize3 = 804
UNCOMP XML w/o streambuf - iterations:10000
time5 = 8546
filesize5 = 3698
UNCOMP bin w/o streambuf - iterations:10000
time6 = 4771
filesize6 = 510
boost is fun ;]
- Attachments
-
- tests.tar.gz
- test to check various compress algorithms time and file size.
- (4.83 KiB) Downloaded 127 times