Multipllayer not possible - Serialization not compatible

Describe your experience with the latest version of FreeOrion to help us improve it.

Moderator: Oberlus

Forum rules
Always mention the exact version of FreeOrion you are testing.

When reporting an issue regarding the AI, if possible provide the relevant AI log file and a save game file that demonstrates the issue.
Post Reply
Message
Author
Ophiuchus
Programmer
Posts: 3433
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Multipllayer not possible - Serialization not compatible

#1 Post by Ophiuchus »

Hi guys,

I'm playing the release build and wanted to introduce my friends to it.
Trouble is that the necessary serialization is not working - we got linux server and windows clients and the server connection dies when I try to connect from a windows client.

None of my friends has a linux system and anyway the UI rendering on windows seems less resource intensive anyway.

When testing I also found that the 64bit and 32bit versions on linux are not compatible.
I saw the decision on using binary serialization -> maybe text-form serialization will have less issues.
I don't think this is so much a problem for save games, but for client-server communication its really a problem.

I still hope there might be a command-line switch for changing that behavior ;)
I'd might be able to help coding, but I would also need to be able to build on windows; which didn't work for me last time I tried.

Thanks,
Snakeguy
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Multipllayer not possible - Serialization not compatible

#2 Post by Dilvish »

Ophiuchus wrote:When testing I also found that the 64bit and 32bit versions on linux are not compatible.
I saw the decision on using binary serialization -> maybe text-form serialization will have less issues.
I don't think this is so much a problem for save games, but for client-server communication its really a problem.
Yes, we've confirmed last year that the text serialization, unlike binary serialization, was cross platform compatible; the issue applies to both server-client communications and to savegame format. In the past the larger size of text serialization had caused problems for some people so it's been left as binary. There has also been some talk of switching to text serialization which is then compressed in a cross-platform compatible way, but it's been backburner item.

The type of archive, xml text vs binary, is currently specified by the

Code: Select all

#define FREEORION_BINARY_SERIALIZATION 1
preprocessor directive in util/serialization.h. So if you can get your build process on windows working you could build your own versions using the text serialization. There have been some improvements/reductions in system requirements for compiling/linking in the past year, so you might want to try again. If you search the forums you can also find some tips for recommended MSVC settings.

Regarding changing it over to being controlled by a runtime option, it does seem to me that it is likely to be do-able, either by making all the Serialize functions be templated or to just declare separate versions for binary and text archives, and then tracking down the places where (de)serialization is initiated and having the type get gated there by a runtime option. If you got that working it would be a great contribution.

Side-note to Vezzra: although it's probably not a good use of time to have you do separate binary and text serialization builds each week, perhaps that is something you could do periodically, perhaps once every couple months, so folks in this situation could have a prebuilt solution?
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

JdH
Space Floater
Posts: 30
Joined: Mon Sep 01, 2014 11:09 am

Re: Multipllayer not possible - Serialization not compatible

#3 Post by JdH »

Have you ever looked into Portable Binary Achive?
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Cjkjvfnby
AI Contributor
Posts: 539
Joined: Tue Jun 24, 2014 9:55 pm

Re: Multipllayer not possible - Serialization not compatible

#4 Post by Cjkjvfnby »

JdH wrote:Have you ever looked into Portable Binary Achive?
IMHO packing text with zip or gzip can greatly reduce text. (zip is already present on all platforms, gzip can be read as stream)
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Multipllayer not possible - Serialization not compatible

#5 Post by Vezzra »

Dilvish wrote:Side-note to Vezzra: although it's probably not a good use of time to have you do separate binary and text serialization builds each week, perhaps that is something you could do periodically, perhaps once every couple months, so folks in this situation could have a prebuilt solution?
Should be doable... although in that case it would be easier if that #define would be done via compiler flag -D.

That said, I can only provide such builds for Mac and Win, not for Linux, which however is what the OP needs. Mac and Win builds are binary compatible anyway, so I wonder how much would really be gained if I provide builds for these platforms with text based serialization?

I'd rather vote for switching back to text based serialization entirely.

User avatar
em3
Vacuum Dragon
Posts: 630
Joined: Sun Sep 25, 2011 2:51 pm

Re: Multipllayer not possible - Serialization not compatible

#6 Post by em3 »

Vezzra wrote:That said, I can only provide such builds for Mac and Win, not for Linux, which however is what the OP needs. Mac and Win builds are binary compatible anyway, so I wonder how much would really be gained if I provide builds for these platforms with text based serialization?
This would help windows/mac guys wanting to play with linux guys.
https://github.com/mmoderau
[...] for Man has earned his right to hold this planet against all comers, by virtue of occasionally producing someone totally batshit insane. - Randall Munroe, title text to xkcd #556

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Multipllayer not possible - Serialization not compatible

#7 Post by Dilvish »

Just to add a little to em3's post-- the server these guys are using is apparently on Linux, & whoever is running that should be able to handle compiling FO themselves, so providing Windows and Mac clients with text serialization *should* solve the problem for them.

Regarding as swapping over entirely to text serialization, my recollection is that the text format takes up enough space that it does noticeably slow things down, for some people significantly. I'll try to double check it on my machine sometime soon.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Multipllayer not possible - Serialization not compatible

#8 Post by Geoff the Medio »

Dilvish wrote:Regarding as swapping over entirely to text serialization, my recollection is that the text format takes up enough space that it does noticeably slow things down, for some people significantly. I'll try to double check it on my machine sometime soon.
The plan is to do so with compression of the serialized text archive. Assuming the compression is portable, that should somewhat reduce the size issue. Doing this requires Boost to be built with support from a compression library such as zlib, which the Windows and Mac SDKs are presently not. This is part of the motivation for the SDK update in the SDK / SDL branch.

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Multipllayer not possible - Serialization not compatible

#9 Post by Vezzra »

Dilvish wrote:Just to add a little to em3's post-- the server these guys are using is apparently on Linux, & whoever is running that should be able to handle compiling FO themselves, so providing Windows and Mac clients with text serialization *should* solve the problem for them.
Ack, in that case, should we also consider providing builds with text serialization for the 0.4.4 release? Just wondering...

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Multipllayer not possible - Serialization not compatible

#10 Post by Dilvish »

ok, well after a quick test on my machine-- timing a 100 turn game with "--auto-advance-n-turns 100 --auto-quit", with text serialization it took twice as long as with binary serialization, which doesn't seem bad.
Vezzra wrote:in that case, should we also consider providing builds with text serialization for the 0.4.4 release? Just wondering...
I guess it is seeming to me like that would be a helpful thing.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Multipllayer not possible - Serialization not compatible

#11 Post by Dilvish »

I decided to go ahead and tackle the problem, just by providing runtime control of serialization format, plus a little bit of savegame read flexibility; that patch can be found here.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

Ophiuchus
Programmer
Posts: 3433
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Multipllayer not possible - Serialization not compatible

#12 Post by Ophiuchus »

I think I forgot to thank you guys for the fix in the past.
Thaaaaaaank you :)
Your work is very much appreciated
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Look, ma... four combat bouts!

Post Reply