New SDKs & transition to C++11

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Message
Author
User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

New SDKs & transition to C++11

#1 Post by Vezzra »

As far as I understand the new SDKs will finally enable us using C++11, so considering that these new SDKs are almost ready I assume this switch is imminent. As MSVC 2013 and the v120 toolset apparently have only rudimentary C++11 support, I guess that also means that support for MSVC 2013 will be deprecated? I'm asking because I want to know if I should/have to upgrade the dev environments on my Win machines to MSVC 2015.

The second thing is the naming scheme for the new SDKs. @Marcel, I've seen that you introduced a version numbering scheme based on the FO version. I'm wondering if that really makes much sense, as the version of the SDK isn't really tied to that of FO itself - if anything, it's tied to a specific point in the commit history, usually because you need a certain version of the SDK to be able to build FO from a certain commit onward, and/or the other way round: a certain version of the SDK can only be used to build FO from a certain commit onward.

Which is why we used to base the SDK version only on the date.

I actually don't have a strong objection against your new scheme, I just don't see the point. If Geoff also thinks it's okay, I'm fine with that - Geoff?

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: New SDKs & transition to C++11

#2 Post by Geoff the Medio »

MSVC 2013 actually supports a fair number of C++11 features, so whether or not MVSC 2015 is required really depends on what C++11 features are desired. I've never had a chance to use any of it, so I can't comment on what those would be...

That said, how difficult would it be for your to switch to MSVC 2015?

Presumably there should be both SDKs labelled for the date and for particular (release) versions of FreeOrion?

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: New SDKs & transition to C++11

#3 Post by adrian_broher »

Vezzra wrote:As far as I understand the new SDKs will finally enable us using C++11, so considering that these new SDKs are almost ready I assume this switch is imminent. As MSVC 2013 and the v120 toolset apparently have only rudimentary C++11 support, I guess that also means that support for MSVC 2013 will be deprecated? I'm asking because I want to know if I should/have to upgrade the dev environments on my Win machines to MSVC 2015.
You can keep Visual Studio 2013 as long as you want but you need to install the v140/v140_xp toolset.
Vezzra wrote:The second thing is the naming scheme for the new SDKs. @Marcel, I've seen that you introduced a version numbering scheme based on the FO version. I'm wondering if that really makes much sense, as the version of the SDK isn't really tied to that of FO itself - if anything, it's tied to a specific point in the commit history, usually because you need a certain version of the SDK to be able to build FO from a certain commit onward, and/or the other way round: a certain version of the SDK can only be used to build FO from a certain commit onward.
When writing the SDK I just realised that tracking the date as the primary version it just adds overhead to sync tag, ChangeLog.md and artifact name. I don't want to update the ChangeLog.md, commit it, tag the release and hope that the build service is in the same timezone and creates the right artifact name. The associated FreeOrion version and a sequence number is easier to handle IME.

The date is completely inaccurate (in relation to what timezone? Oh great, the local committer one!) and doesn't map the commit and the SDK change. Nothing hinders me to merge commits I wrote a year ago and require a new SDK version (Author and committer dates don't change when merging/pushing). Nothing hinders me to commit a hundred commits that day and require a change for an SDK somewhere in between. If you want to solve this problem definitely you need to add a tag, that indicates the need for a new SDK like 'with-FreeOrionSDK-v0.4.6-1'.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: New SDKs & transition to C++11

#4 Post by adrian_broher »

Geoff the Medio wrote:MSVC 2013 actually supports a fair number of C++11 features, so whether or not MVSC 2015 is required really depends on what C++11 features are desired.
Alle.

=3
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: New SDKs & transition to C++11

#5 Post by adrian_broher »

So is there any estimated date you can switch your build machine? Because that's the only thing left to do, right?
  • The SDKs are build against the system libraries required to support C++11.
  • All build systems are updated to support C++11 and build against the new SDKs.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#6 Post by Vezzra »

adrian_broher wrote:So is there any estimated date you can switch your build machine?
In theory I should have been able to be done with that by today, alas, this weird, hard to catch crash disrupted my plans and has been eating too much of my FO time these last few days.

I hope to get that done next weekend.
Because that's the only thing left to do, right?
Almost. There is one thing left I'd like to check, and that is if the binaries produced with the new SDK and C++11 enabled actually do run on a 10.7 system. And if not, on which it does. I can probably live with only supporting 10.8 and later, although I'm not thrilled by that perspective, but if it's only 10.9 or even only 10.10 and later, that would be bad.

The challenge here being getting access to a 10.7 or 10.8 system. Haven't figured out that yet.

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#7 Post by Vezzra »

adrian_broher wrote:The date is completely inaccurate (in relation to what timezone? Oh great, the local committer one!) and doesn't map the commit and the SDK change. Nothing hinders me to merge commits I wrote a year ago and require a new SDK version (Author and committer dates don't change when merging/pushing). Nothing hinders me to commit a hundred commits that day and require a change for an SDK somewhere in between.
I understand completely, and am certainly not married to using the date as "version number" for the SDK. This has just been a convenient way to come up with a number (IMO).

We can just as well use an arbitrary number we increase each time a new version of the SDK needs to be released. The reason I'm a bit wary about the idea to correlate FO's and the SDK's version numbers is that it's potentially a bit misleading. The need for a new SDK is in no way connected to the current release we're working on, in theory we can go a couple of releases without needing a new SDK.

However, having a "SDK-v0.4.6-1" somehow indicates that this SDK is needed to build FO 0.4.6. Which might, or better, in most cases will be, wrong. The need for a new SDK will almost always arise while we're between two releases. Which means, the current FO version number is "v<last-release-version>+". Following your scheme, the new SDK's version number will be ""v<last-release-version>-1". But that SDK will be needed to build FO versions after the last release, and the last release very likely might not be buildable with the new SDK.

And there is certainly no point in releasing a new SDK just because a new version of FO has been released, just to keep the version numbers in sync.

For these reasons I'd prefer to decouple FO's and the SDK's version numbering.
If you want to solve this problem definitely you need to add a tag, that indicates the need for a new SDK like 'with-FreeOrionSDK-v0.4.6-1'.
That certainly a very valid point. I second that suggestion, proposing to employ that policy already with the new SDK once we officially make the switch.

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#8 Post by Vezzra »

Vezzra wrote:
adrian_broher wrote:So is there any estimated date you can switch your build machine?
[...]

I hope to get that done next weekend.
Yeah, so much for my plans - sorry. I've got good and bad news: the good one being I got around to install MSVC 2015 on my Windows machine, and managed to build FO successfully with it. The bad news are, that when I tried to run that build on my XP machine, the server crashes (apparently during or after Python initialization). Problem: as I can't run FO on the machine I build it on, and can't build FO on the machine I can run it on, I'm at a loss on how to get a backtrace, which I guess is very much needed to troubleshoot the issue.

Well, after this incident I finally had enough, and decided to upgrade that decrepit dinosaur-esque excuse of a computer at least to the point where I'm able to build FO in under 6 hours and also to run it on that same machine. The required parts have already been ordered, hopefully I'll get them at the end of this or during the next week, so should be up and running in two weeks.

Until then I can only offer very meager infos about the crash: I try to start a default quickstart game, and after a few moments get an error message: "Timed out while attempting to connect to server". Here are the entire contents of freeoriond.log:

Code: Select all

2016-12-02 15:59:23.203250 [debug] Server : Logger.cpp:67 : Logger initialized
2016-12-02 15:59:23.203250 [info] Server : Logger.cpp:68 : v0.4.6+ add_cpp11 [build 2016-11-25.9e6b9cb] MSVC 2015
2016-12-02 15:59:23.203250 [info] Server : DependencyVersions.cpp:82 : Dependency versions from headers:
2016-12-02 15:59:23.203250 [info] Server : DependencyVersions.cpp:87 : Boost: 1_59
2016-12-02 15:59:23.203250 [info] Server : DependencyVersions.cpp:87 : Python: 2.7.12
2016-12-02 15:59:23.203250 [info] Server : DependencyVersions.cpp:87 : zlib: 1.2.8
2016-12-02 15:59:23.203250 [debug] Server : ServerFSM.cpp:256 : (ServerFSM) Idle
2016-12-02 15:59:23.203250 [debug] Server : CommonFramework.cpp:38 : Initializing FreeOrion Python interface
2016-12-02 15:59:23.203250 [debug] Server : CommonFramework.cpp:49 : Python home set to C:\Programme\FreeOrion
2016-12-02 15:59:23.203250 [debug] Server : CommonFramework.cpp:52 : Python program name set to C:\Programme\FreeOrion\freeoriond.exe
There is one thing left I'd like to check, and that is if the binaries produced with the new SDK and C++11 enabled actually do run on a 10.7 system. And if not, on which it does. I can probably live with only supporting 10.8 and later, although I'm not thrilled by that perspective, but if it's only 10.9 or even only 10.10 and later, that would be bad.

The challenge here being getting access to a 10.7 or 10.8 system. Haven't figured out that yet.
Still stuck on that one too, so I have to ask for your patience...

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#9 Post by Vezzra »

Vezzra wrote:The challenge here being getting access to a 10.7 or 10.8 system. Haven't figured out that yet.
Found an install image for 10.8 and have been able to set up a 10.8 system. Unfortunately I have some bad news: although an FO build produced with the new SDK and C++11 enabled can be launched and a game started, executing a turn results in a crash - apparently the AI clients crash when trying to deserialize the game state. This happens regardless if binary serialization is turned on or not. The build I used is the add_cpp11 branch rebased onto current master.

The build runs fine on OSX 10.9.

A build of current master produced with the old SDK and without C++11 support runs fine on 10.8

Attached are the log files for the crashes on 10.8 (two tries with XML and binary serialization on 10.8 ).
Attachments
AI_Crash_10.8_Binary_Serialization.zip
(163.05 KiB) Downloaded 170 times
AI_Crash_10.8_XML_Serialization.zip
(124.53 KiB) Downloaded 157 times

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: New SDKs & transition to C++11

#10 Post by adrian_broher »

Attached are the log files for the crashes on 10.8 (two tires with XML and binary serialization on 10.8).
Both configurations contain the message

Code: Select all

Error: unrecognized XML syntax
Which is exclusive for the xml variant of the boost serialization library. So you never actually run the binary serialization on the clients or the server, regardless of setting?

Well, try to get a dump of the network communication between server and clients to check if this is actually the case? tcpdump or wireshark should help here.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: New SDKs & transition to C++11

#11 Post by Geoff the Medio »

adrian_broher wrote:Which is exclusive for the xml variant of the boost serialization library. So you never actually run the binary serialization on the clients or the server, regardless of setting?
It always tries both when deserializing... binary first then xml if binary failed. The setting just changes the "forward" serialization.

LGM-Doyle
Programmer
Posts: 219
Joined: Mon Feb 29, 2016 8:37 pm

Re: New SDKs & transition to C++11

#12 Post by LGM-Doyle »

Vezzra in both versions of your freeoriond.log file the earliest error is

Code: Select all

2016-12-04 17:55:44.179058 [error] Server : Effect.cpp:462 : SetMeter::Execute couldn't cast simple increment ValueRef to an Operation. Reverting to standard execute.
This occurs before the serialization problems in both sets of data, so I think that the state is corrupted even before serialization is attempted.

The code at Effect.cpp:462 is:

Code: Select all

        ValueRef::Operation<double>* op = dynamic_cast<ValueRef::Operation<double>*>(m_value);
        if (!op) {
            ErrorLogger() << "SetMeter::Execute couldn't cast simple increment ValueRef to an Operation. Reverting to standard execute.";
            EffectBase::Execute(context, targets);
            return;
        }
I don't know what the problem is. I would start the game in the debugger and set a break point on Effect.cpp:462 and then examine m_value.

Hopefully, m_value is already obviously bad at that point and you can walk bad, until you find where it went bad.

If m_value is not obviously bad prior to the dynamic_cast, then check if the dynamic_cast is trying an implicit narrowing conversion. The rules for these were made stricter in C++11 and we may need to provide explicit conversions for whatever type of m_value is failing here.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: New SDKs & transition to C++11

#13 Post by Geoff the Medio »

LGM-Doyle wrote:If m_value is not obviously bad prior to the dynamic_cast, then check if the dynamic_cast is trying an implicit narrowing conversion. The rules for these were made stricter in C++11 and we may need to provide explicit conversions for whatever type of m_value is failing here.
m_value has type ValueRefBase<double>* and Operation<double> has the base class ValueRefBase<double>. ValueRefBase<T> has only one constructor with no parameters. How is implicit narrowing possible here?

LGM-Doyle
Programmer
Posts: 219
Joined: Mon Feb 29, 2016 8:37 pm

Re: New SDKs & transition to C++11

#14 Post by LGM-Doyle »

I am only guessing. I should have stopped at I'd start up the debugger and set a break point.

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: New SDKs & transition to C++11

#15 Post by Vezzra »

Sorry, guys, for my lack of responses the last days, but there has been a reason for it (of course), and I've also got some good news :D

The parts for my computer upgrade arrived very quickly, and I've been busy dis- and reassembling my main Windows/Linux dev machine and completely reinstalling both OSes from scratch. Of course ran into a couple of issues (nothing ever goes perfectly smoothly ;)) which had to be solved, but finally I'm up and running again - big hooray!

I then built the new SDK with the upgraded system, and built the add_cpp11 branch, rebased onto current (commit 450727aa) master, with that new SDK. That build ran successfully on the dev machine (finally I'm able to run FO on the Windows machine I build it on!), and, surprise, it also ran successfully on my old XP machine. No server crash like before. I don't know what changed, maybe I did something wrong last time, but hey, it works now.

@Marcel: This means we're practically ready to do the switch to the new SDK. Concerning the issue with OSX builds crashing on 10.8, as much as it bothers me, I don't want this to delay the switch to the new SDK (and with that the switch to enabling C++11 support) any longer. We can (and should of course) continue to investigate and work on that issue. If we can solve it and thus be able to support OSX versions prior to 10.9, all the better, if not, well, I guess I can live with that (hopefully there are not that many people left out there who are still on 10.8 or earlier).

Points I'd like to address before we do the actual switch:

1.) The text boostrap.bat script prints to the console still only mentions MSVC 2013. Either simply change that to MSVC 2015, or mention both (I have a mild preference for the former). Besides, this raises the question: do we want to keep support for MSVC 2013? Is that even possible, if I understood correctly, we need MSVC 2015 anyway for C++11 support? Geoff?

2.) Version numbering scheme for the SDK: I perfectly ok with dropping any date references, but as I mentioned above, I don't think deriving the SDK version from the concurrent FO version is a good idea. Is there any reason not to just simply have an independent version number for the SDK?

Post Reply