New SDKs & transition to C++11

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Message
Author
User avatar
adrian_broher
Programmer
Posts: 1072
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: New SDKs & transition to C++11

#16 Post by adrian_broher » Sun Dec 11, 2016 9:22 am

The text boostrap.bat script prints to the console still only mentions MSVC 2013. Either simply change that to MSVC 2015, or mention both (I have a mild preference for the former). Besides, this raises the question: do we want to keep support for MSVC 2013? Is that even possible, if I understood correctly, we need MSVC 2015 anyway for C++11 support? Geoff?
Isn't the same true for the bootstrap.command?

Code: Select all

echo 'Note: You need to have Xcode 5 or later installed to use the SDK'
How is the relation between Xcode and OSX Platform SDK? Afaik there is none and Apple encourages you to use the latest Xcode version.
Version numbering scheme for the SDK: I perfectly ok with dropping any date references, but as I mentioned above, I don't think deriving the SDK version from the concurrent FO version is a good idea. Is there any reason not to just simply have an independent version number for the SDK?
I don't want to bother with this anymore, so a sequence number it is (v1, v2, v3, …).
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12249
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: New SDKs & transition to C++11

#17 Post by Geoff the Medio » Sun Dec 11, 2016 9:51 am

Vezzra wrote:...do we want to keep support for MSVC 2013? Is that even possible, if I understood correctly, we need MSVC 2015 anyway for C++11 support? Geoff?
For some features yes, others no.

https://msdn.microsoft.com/en-us/library/hh567368.aspx

Are we sure that going to 2015 doesn't break support for Windows XP?

User avatar
Vezzra
Release Manager, Design
Posts: 4610
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#18 Post by Vezzra » Sun Dec 11, 2016 2:45 pm

adrian_broher wrote:Isn't the same true for the bootstrap.command?
Yep, you're right. Just ran a few tests and found out that Xcode 5.1.1 (the latest release of Xcode 5) requires OSX 10.10 anyway (I'm not going through the hassle of finding out if earlier releases of Xcode 5 work on 10.8 or 10.9), which is the system I'm on. The latest version of Xcode that runs on 10.10 is Xcode 6.4, so I'm going to make this easy and hereby raise the minimum requirements to use the SDK to OSX 10.10 and Xcode 6.4.

This much I can guarantee, as it's the exact configuration I'm using right now, and which I know works.

I've updated bootstrap.command accordingly (a5c5b10)
How is the relation between Xcode and OSX Platform SDK? Afaik there is none and Apple encourages you to use the latest Xcode version.
Well, the OSX platform SDKs ship with the Xcode app, that's the only relation I know of. And I guess any Xcode version can only support the SDKs it contains and earlier ones.

Using the latest Xcode version available isn't always possible, that depends on the OSX version you're on. I, for example, can't use Xcode 7 or later, as Xcode 7 requires at least OSX 10.11. And updating to the latest OSX version isn't always a good idea either, or even possible (depending how old your computer is). So I don't recommend to require the latest OSX and Xcode versions to be able to use the SDK.
I don't want to bother with this anymore, so a sequence number it is (v1, v2, v3, …).
Sorry, didn't want to be annoying about that... :oops:

User avatar
Vezzra
Release Manager, Design
Posts: 4610
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#19 Post by Vezzra » Sun Dec 11, 2016 3:08 pm

Geoff the Medio wrote:For some features yes, others no.
Ok, I see. Well, your call. If you want to keep support for MSVC 2013 for the time being, that's fine with me.
Are we sure that going to 2015 doesn't break support for Windows XP?
Pretty much. Quoting myself:
Vezzra wrote:I then built the new SDK with the upgraded system, and built the add_cpp11 branch, rebased onto current (commit 450727aa) master, with that new SDK. That build ran successfully on the dev machine (finally I'm able to run FO on the Windows machine I build it on!), and, surprise, it also ran successfully on my old XP machine. No server crash like before. I don't know what changed, maybe I did something wrong last time, but hey, it works now.
In case it isn't clear: I produced that build with MSVC 2015 (have never been able to build the add_cpp11 branch with MSVC 2013). So a build of the add_cpp1 branch produced with MSVC 2015 indeed does run on XP.

Unless MSVC mixes things up when both MSVC 2013 and 2015 are installed on the same machine. I'm relying on the assumption that when I open the MSVC 2015 solution in MSVC 2015 and build FO, that MSVC 2015 actually uses exclusively it's own v140 toolchain, and nothing else...?

User avatar
Vezzra
Release Manager, Design
Posts: 4610
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#20 Post by Vezzra » Sun Dec 11, 2016 3:56 pm

adrian_broher wrote:Well, try to get a dump of the network communication between server and clients to check if this is actually the case? tcpdump or wireshark should help here.
Ok, did that, but I'm afraid I can't make head or tail of it. I switched between binary and XML serialization, but I can't even tell if that makes any difference... when looking at the TCP streams, there is definitely XML stuff in there, even when switching to binary. However, as that's to be expected (as even binary serialization has XML headers if I understand correctly), that doesn't tell much.

There are definitely huge chunks which don't look like XML at all, but again, that's the case regardless what the serialization mode is. And that's to be expected too, as XML serialization compresses the game state. However, in that case I'd expect to only look at binary gibberish, but I definitely can read a lot of strings in those huge non-XML chunks... which should not be possible with XML serialization, right?

Sorry, all I can offer is my sincere confusion... :(

User avatar
Vezzra
Release Manager, Design
Posts: 4610
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#21 Post by Vezzra » Sun Dec 11, 2016 3:59 pm

Geoff the Medio wrote:The setting just changes the "forward" serialization.
What's "forward" serialization?

User avatar
Vezzra
Release Manager, Design
Posts: 4610
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#22 Post by Vezzra » Sun Dec 11, 2016 4:25 pm

LGM-Doyle wrote:Vezzra in both versions of your freeoriond.log file the earliest error is

Code: Select all

2016-12-04 17:55:44.179058 [error] Server : Effect.cpp:462 : SetMeter::Execute couldn't cast simple increment ValueRef to an Operation. Reverting to standard execute.
This occurs before the serialization problems in both sets of data, so I think that the state is corrupted even before serialization is attempted.
Right, and this error shows up quite a lot. Even in some of the AI logs. This most likely indicates that something is seriously amiss...
I would start the game in the debugger and set a break point on Effect.cpp:462 and then examine m_value.
Um, I think we've got a problem here - as I said, this issue only occurs on 10.8, and the 10.8 system I use is a very barebone installation on an external hard drive, with no dev environment, any debug tools or whatever. The Xcode version I use to build FO can't even be installed on 10.8 (because it requires 10.10), and I highly doubt that a Xcode 4 dev environment will allow me to debug a build produced with Xcode 6.

Not to speak of the problem that I've never been able to launch FO from within Xcode anyway.

So I'm afraid if we want to track that nasty down, we'll have it to do the old-fashioned, mind-numbingly hard way: insert hard coded debug output, build, reboot into 10.8, run, reboot into 10.10, analyze debug output, insert some more hard coded debug output, lather, rinse, repeat until I find something useful or go mad, whichever comes first... :?

Ugh. I don't know if I want to do this... that sounds awfully time-consuming and frustrating. Why can't things just work???? :(

What does/should m_value contain? Can I just do "... << m_value", or is something else required?

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12249
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: New SDKs & transition to C++11

#23 Post by Geoff the Medio » Sun Dec 11, 2016 4:44 pm

Vezzra wrote:I switched between binary and XML serialization, but I can't even tell if that makes any difference... when looking at the TCP streams, there is definitely XML stuff in there, even when switching to binary. However, as that's to be expected (as even binary serialization has XML headers if I understand correctly), that doesn't tell much.
The binary serialization option in the GUI only controls save files. For client-server messages, it's based on comparing version strings for a few cases, like turn update messages that are potentially very large, and just uses (uncompressed) XML for most / all other smaller messages, which are generally small enough to have little reason to need less-verbose serialization.

https://github.com/freeorion/freeorion/ ... .cpp#L3211

User avatar
Vezzra
Release Manager, Design
Posts: 4610
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#24 Post by Vezzra » Sun Dec 11, 2016 4:48 pm

Ah, ok, thanks, that cleared the confusion for me. :D

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12249
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: New SDKs & transition to C++11

#25 Post by Geoff the Medio » Sun Dec 11, 2016 4:50 pm

Vezzra wrote:What does/should m_value contain? Can I just do "... << m_value", or is something else required?
Probably ... << m_value->Dump(). If we're talking about SetMeter::Execute, m_value is the ValueRef that returns a number which specifies what to set the meter to. All ValueRefs (amongst other similar classes) have a Dump() function that should return text that describes their structure / contents.
Vezzra wrote:
Geoff the Medio wrote:The setting just changes the "forward" serialization.
What's "forward" serialization?
Taking data structures in memory and converting it to something that can be saved to disk or sent by the network, such as XML, to store or transmit the original data. Point is, the save serialization option only affects what is saved, not what it does when it loads. It always tries both modes when loading, so it should be able to read any compatible-version save file, regardless of what mode it was saved it.

User avatar
Vezzra
Release Manager, Design
Posts: 4610
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#26 Post by Vezzra » Sun Dec 18, 2016 4:58 pm

With V3 of the SDK as the first "official" release of the new SDK I did some more tests, this time I tried using the SDKs you can download from github (which get built and released automatically by Travis and AppVeyor when you tag a commit in the SDK repo if I understand correctly). That worked flawlessly on OSX, on Windows however I got a strange build error for the Common project. The error occurs when trying to execute the make_versioncpp.py script, cmd.exe exits with a cryptic error code.

Upon investigation I found out that the python.exe in the bin folder of the SDK apparently is different from the one I get when building the SDK myself on my system. This python.exe built by AppVeyor requires a LIBEAY32.dll and fails to execute if that DLL can't be found. The python.exe I get when building the SDK on my system executes without any problem.

@Marcel: any idea what could be different in the AppVeyor environment that causes this difference? @Geoff: can you test the SDK provided on github, if the issue turns up on your system too?

Another (minor) thing: there is a libpython.py in the bin folder of the Windows SDK, what's the purpose of this Python module? It hasn't been present in the old SDK, and apparently wasn't required. I noticed it because it showed up in my git client as an unversioned file. If it's not required, can we remove it? Otherwise we should include it in .gitignore.

User avatar
Vezzra
Release Manager, Design
Posts: 4610
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#27 Post by Vezzra » Sun Dec 18, 2016 5:07 pm

@Marcel: aside from these last bits, is there anything else left you want/need to do/address regarding the new SDK, or can we/I go ahead and update all the relevant wiki pages and forum threads for use of the new SDK?

Another thing is the old SDK download location on sourceforge: that needs updating too, I suggest moving the old SDKs into the "Old versions" folder and only leaving a ReadMe.txt in the SDK download folder providing a short notice redirecting anyone who looks for SDKs there to github. Is that ok with everyone? Anyone any objections?

User avatar
adrian_broher
Programmer
Posts: 1072
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: New SDKs & transition to C++11

#28 Post by adrian_broher » Sun Dec 18, 2016 6:19 pm

Vezzra wrote:@Marcel: any idea what could be different in the AppVeyor environment that causes this difference?
Seems to be that library is related to OpenSSL: http://stackoverflow.com/questions/6534 ... ound-error

I assume AppVeyor has OpenSSL installed (confirmed) and the CMake configuration for the Python build picks up OpenSSL when not explicit disabled.
Vezzra wrote:Another (minor) thing: there is a libpython.py in the bin folder of the Windows SDK, what's the purpose of this Python module? It hasn't been present in the old SDK, and apparently wasn't required. I noticed it because it showed up in my git client as an unversioned file. If it's not required, can we remove it? Otherwise we should include it in .gitignore.
No that is not required and we should remove it in the post install python step of the SDK build.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
adrian_broher
Programmer
Posts: 1072
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: New SDKs & transition to C++11

#29 Post by adrian_broher » Sun Dec 18, 2016 11:26 pm

You download the new SDK artifact with an updated configuration as soon as this build is ready:

https://ci.appveyor.com/project/freeori ... /artifacts
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
adrian_broher
Programmer
Posts: 1072
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: New SDKs & transition to C++11

#30 Post by adrian_broher » Mon Dec 19, 2016 8:53 am

Vezzra wrote:what's the purpose of this Python module?
It's not a module but some convenience wrapper around libpython/python to hook gdb into a running process. python-cmake-buildsystem installs this unconditionally.
Vezzra wrote:sAide from these last bits, is there anything else left you want/need to do/address regarding the new SDK, or can we/I go ahead and update all the relevant wiki pages and forum threads for use of the new SDK?
No, there is nothing to address from my side. I already updated some of the Wiki pages that mention the SDK and updated those to the new SDK download location.
Vezzra wrote:Another thing is the old SDK download location on sourceforge: that needs updating too, I suggest moving the old SDKs into the "Old versions" folder and only leaving a ReadMe.txt in the SDK download folder providing a short notice redirecting anyone who looks for SDKs there to github. Is that ok with everyone? Anyone any objections?
No, I haven't thought of that and it's a great idea.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

Post Reply