CMakeify the SDK build process.

Discussion about the project in general, organization, website, or any other details that aren't directly about the game.
Message
Author
User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

CMakeify the SDK build process.

#1 Post by adrian_broher »

You may wonder what's going on with that freeorion/freeorion-sdk repository I created today. Well, I started working some time ago on coding the build process of the FreeOrion SDK. The and reasons for this are manifold:

The process should be documented in a central place
To fix changes or suggest improvements it's always better to actually know how something is done. The SDK creation is currently a black box process, which makes it hard to suggest improvements.

There should not be a single point of responsibility
The Windows SDK is maintained by Geoff, the MacOSX one by Vezzra. It happened several times that both were not available for for correcting errors or update the SDK, which is understandable given the nature of this project.

The process should be repeatable
In case the maintainer HDD dies he would need rebuild the SDK from scratch, maybe introducing new bugs. When encoding the build process in a build script it's just a matter of cloning the SDK and running the build script.

The process should be automatable
To be honest it's a huge time sink to build the SDK for the maintainer. That time would be invested better in creating a better game. When using a build script we could utilize build services like Travis or AppVeyor to let them build and release the SDK for us.


I have chosen my favorite workhorse CMake with the last two points in mind because it allows a uniform way to describe the build. Currently the build script works on Linux and mimics the Windows SDK layout from some month ago. The next days I will invest some time in building a proper clone of the current Windows SDK. I will write a comment when it is ready to test. Because I don't have a Mac available I need rely on Vezzra for some feedback here.
Last edited by adrian_broher on Mon Mar 28, 2016 5:38 pm, edited 1 time in total.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: CMakeify the SDK build process.

#2 Post by Vezzra »

Correct me if I'm wrong, but AFAIK what cmake does is produce the required makefiles/project files for whatever IDE which can then be used to build the project, right?

So, what we're talking about here is replacing the MSVC and Xcode projects currently maintained in the repo with cmake scripts, which then can be used to produce the MSVC/Xcode/whatever build system/IDE someone might want to use and is supported by cmake. Thus eleminating the need to maintain cmake scripts for building on Linux, a Xcode project for building on OSX, and a MSVC project for building on Windows.

That's certainly a very good idea (although that means I'll finally have to really wrap my head around cmake, ugh), as it will simplify maintainence of the build system.

However, if we are talking about the SDKs, these are actually something different. The SDKs are just packages that contain the prebuilt dependencies for the Windows and OSX platforms (something which isn't necessary on Linux). Or can you really set up cmake scripts that can actually fetch (and, where necessary, build) all those dependencies and produce the SDK package? That of course would be beyond cool...

Otherwise, if what you're referring to when you say "SDK" is actually not the package with the prebuilt dependencies, but the MSVC/Xcode project, then using the term "SDK" for them is a bit misleading. ;)

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: CMakeify the SDK build process.

#3 Post by adrian_broher »

Vezzra wrote:Correct me if I'm wrong.
You're wrong, please let me correct you. :wink:
Vezzra wrote:, but AFAIK what cmake does is produce the required makefiles/project files for whatever IDE which can then be used to build the project, right?
That is correct, but cmake can do more than that.
Vezzra wrote:So, what we're talking about here is replacing the MSVC and Xcode projects currently maintained in the repo with cmake scripts, which then can be used to produce the MSVC/Xcode/whatever build system/IDE someone might want to use and is supported by cmake. Thus eleminating the need to maintain cmake scripts for building on Linux, a Xcode project for building on OSX, and a MSVC project for building on Windows.
No, that is not what I am talking about here. While it is technical possible (and preferable for the reasons you mentioned) to replace the hand written project files with autogenerated project files in the FreeOrion Project this thread is about the SDK we use to distribute the precompiled dependencies (python, boost, libpng, libvorbis¸ …) to the platforms that don't have a package management system/software (Window, Mac OSX) available.
Vezzra wrote:However, if we are talking about the SDKs, these are actually something different. The SDKs are just packages that contain the prebuilt dependencies for the Windows and OSX platforms (something which isn't necessary on Linux). Or can you really set up cmake scripts that can actually fetch (and, where necessary, build) all those dependencies and produce the SDK package? That of course would be beyond cool...
The second last sentence is the goal I had in mind and while I'm not yet there I'm pretty close to realizing this. The build script does already the tasks you mentioned. For every dependency it downloads a specified source release, configure, build and 'installs' it into a directory that can be packaged into a SDK file. This packaging isn't done yet but adding a zip task isn't too hard to realize.


However I'm still checking if the dependencies are built properly on Windows. Currently I'm lacking for a proper test environment (Windows systems are rare in this household and paying alot of money for a OS I usually don't use is not really my intention), but I'm checking out if I can utilize the virtual machines provided by Microsoft for IE web page testing as dev boxes.

When I've done that and tweaked the CMake script accordingly I can think about packaging the 'install' directory into something that is equal to the currently released SDK file.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: CMakeify the SDK build process.

#4 Post by Vezzra »

That definitely sounds very intriguing. If you can really pull that off, putting together cmake scripts that gather, download and build all the dependencies and produce the SDK packages with the required directory structure and the bootstrap scripts, that will certainly make maintaining those SDKs much easier. :D

Although I know that especially on Windows building all those dependencies is a pain, Geoff more than once had to overcome apparently quite nasty issues when he needed to update the SDK (e.g. because of switching to a new boost version etc.). I had my share of issues on OSX as well, I think almost every time when I updated the SDK I had to fight some things that wouldn't work like they should.

Well, anyway, it most certainly is worth a try. As I have managed to set up dev environments with all three platforms (Windows, OSX and Linux) on physical machines in the meantime, I can offer to run tests on any of them (although my dual boot Windows/Linux machine is ancient and sloooow).

FYI, here is the shell script I use to build boost for FO on OSX:

Code: Select all

#!/bin/sh
ZLIB_SOURCE=/Users/user/SoftwareProjekte/Libraries/zlib/zlib-1.2.7
./bootstrap.sh
./b2 --clean-all
./b2 -q -a macosx-version-min=10.7 architecture=x86 variant=release link=static threading=multi stage
The last line is probably the most interesting for you. The macosx-version-min option particularly is important, for whatever reason boost wouldn't build properly on my system without it.

One thing we could try when moving to cmake scripts to build the SDKs is to switch to full 64 bit builds. The only reason why I still stick with 32 bit on OSX is that I failed making 64 bit builds so far. Mostly because I don't build the dependencies myself, but use prebuilt binaries whenever ones are provided (SDL, Python, etc.), and IIRC I didn't get 64 bit binaries for all dependencies. Building them myself failed, and I didn't want to waste too much time with that, so I just went with 32 bit until now.

But if the cmake scripts actually builds all the dependencies, then maybe we can get 64 bit builds working. That would be nice.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: CMakeify the SDK build process.

#5 Post by adrian_broher »

Status update:

Building works for windows and creates a SDK zip file. The significant differences between the current and the new Windows SDK are:
  • *.dll and *.exe files are located in the bin and not in the temp subdirectory. (Resolved: bootstrap.bat was adapted to take that in consideration.)
  • The boost version was bumped from 1.58 to 1.59 (this was required because AppVeyor, the CI service, has a slightly different version of Visual Studio 2013 installed. That version complaints about boost.coroutine exposing a class as dll interface that contains thread local storage. I could probably patch 1.58 to do the same but bumping the version was the easier way to go.)
  • Boost now actually depends on the zlib dependency and not their own in-tree version of zlib.
  • The GLEW version was bumped from 1.12 to 1.13 (the old and new GLEW upstream version provide both a cmake build system, but the 1.12 was just horrible broken so I bumped the dependency version instead of backport the new build system. The library didn't changed that much in between.)
  • FreeImage.{lib,dll}, ALUT.{lib,dll} and z.{lib,dll} are not part of the SDK anymore. (I they were not used anyway. Also I have no idea where the z.dll comes from. zlib upstream calls their libraries zlib{.lib,1.dll})
  • The use upstream zlib dll name zlib1.dll
  • All libraries provide a *.pdb file now, even the boost ones. Those are located inside the lib directory.
I must say the most annoying part of creating the script was waiting for the completed compilation. Right after that comes the boost build system. Everything else was pretty much straight forward and when asking the different upstreams for help they replied quickly and competently.

Another thing that now works out of the box (but is disabled for now) is the compilation of the Windows SDK with help of the AppVeyor CI service. Whenever creating an annotated commit AppVeyor will build and create a Windows Release of the SDK on Github. I intent to do the same for Mac with Travis-CI.

Now I just need to set up the same for Mac OSX. Vezzra, what do you think would be the best way to cooperate on this?
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: CMakeify the SDK build process.

#6 Post by Vezzra »

adrian_broher wrote:Vezzra, what do you think would be the best way to cooperate on this?
For starters, you telling me exactly what you need from me, or need me to do, and I will try to comply with those requests asap and as best as I can. :) Keep in mind that I have only an extremely rudimentary understanding of how the things work that you do here, so most likely you're going to have a lot of explaining to do... ;)

For example, what exactly are those AppVeyor and Travis CI services? Could they also be used to automate producing the weekly test builds...?

On a sidenote: feel free to completely revamp the layout of the "dep" folder containing all the prebuilt dependencies and their includes. A long time ago, when I had no idea about adhering to upstream I changed the layout of that folder because the original one didn't seem logical to me.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: CMakeify the SDK build process.

#7 Post by adrian_broher »

Vezzra wrote:For starters, you telling me exactly what you need from me, or need me to do, and I will try to comply with those requests asap and as best as I can. :)
Essentially you need to run the cmake script located in the freeorion-sdk, report errors if there were any, run the generated build system, report errors if there were any, and then create a report of what is different between the currently used SDK and the one created by the script. You can start by reading the README.md (will add some instructions shortly) of https://github.com/freeorion/freeorion-sdk and also cloning that repository. If those instructions are not enough we can continue from that by creating an 'you write issues on the repository -> I provide fixes' cycle until the SDK generation works as expected.
Vezzra wrote:Keep in mind that I have only an extremely rudimentary understanding of how the things work that you do here, so most likely you're going to have a lot of explaining to do... ;)
No problem.
Vezzra wrote:For example, what exactly are those AppVeyor and Travis CI services?
We already utilize both for freeorion. CI stands for Continuous Integration which describes a programming practices to integrate changes by individual developers into some central location (in our case that the git repository) and doing some checking on the whole new product, most simple one being compiling the product. Compiling the changes and checking if the compile was successful is of course unnecessary manual labour, so why don't let a software compile and check every commit? That's what a CI-Servers does. Both AppVeyor and Travis-CI are companies that provides CI-Servers for free for OpenSource projects, if the build time takes less than 60 minutes.
Vezzra wrote:Could they also be used to automate producing the weekly test builds...?
In theory yes, but I have the feeling that the current build time will scratch or even exceed the 60 minutes time limit.
Vezzra wrote:On a sidenote: feel free to completely revamp the layout of the "dep" folder containing all the prebuilt dependencies and their includes. A long time ago, when I had no idea about adhering to upstream I changed the layout of that folder because the original one didn't seem logical to me.
For now I want to keep the SDK layout as is and fix things step by step after successfully replaced the manual with the generated SDK.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: CMakeify the SDK build process.

#8 Post by Vezzra »

Well, just went ahead, cloned the freeorion-sdk repo and ran a test. CMake ran successfully, the subsequent make however failed. I zipped up the entire make output and attached it. Although you probably only need the last few lines:

Code: Select all

[ 33%] Performing build step for 'boost'
Warning: "using python" expects a two part (major, minor) version number; got 2.7.9 instead
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/tools/build/src/build/feature.jam:326: in validate-feature from module feature
error: unknown feature "<debug-store>"
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/tools/build/src/build/feature.jam:360: in expand-subfeatures-aux from module feature
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/tools/build/src/build/feature.jam:425: in feature.expand-subfeatures from module feature
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/tools/build/src/build/build-request.jam:20: in apply-to-property-set from module build-request
(builtin):-1: in sequence.transform from module sequence
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/tools/build/src/build/build-request.jam:32: in build-request.expand-no-defaults from module build-request
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/tools/build/src/build-system.jam:594: in load from module build-system
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/tools/build/src/kernel/modules.jam:295: in import from module modules
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/tools/build/src/kernel/bootstrap.jam:139: in boost-build from module
/Users/user/SoftwareProjekte/FO/sdk/build/boost-prefix/src/boost/boost-build.jam:17: in module scope from module

make[2]: *** [boost-prefix/src/boost-stamp/boost-build] Error 1
make[1]: *** [CMakeFiles/boost.dir/all] Error 2
make: *** [all] Error 2
Attachments
make_ouput.txt.zip
(31.94 KiB) Downloaded 205 times

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: CMakeify the SDK build process.

#9 Post by adrian_broher »

I updated the README anyway. Could you please review it for completeness and comprehensibility?

Also I set up a `mac-support` branch for this whole task in the repository. And I think the issue reporting is probably done best on the repo issue tracker (https://github.com/freeorion/freeorion-sdk/issues).
the subsequent make however failed.
I will take a look at it.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: CMakeify the SDK build process.

#10 Post by adrian_broher »

Yeah, fixed that one with a8fbff1.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

dbenage-cx
Programmer
Posts: 389
Joined: Sun Feb 14, 2016 12:08 am

Re: CMakeify the SDK build process.

#11 Post by dbenage-cx »

Would you want a fresh install perspective now, or wait till issues are sorted with old sdk?
Any content posted should be considered licensed GNU GPL 2.0 and/or CC-BY-SA 3.0 as appropriate.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: CMakeify the SDK build process.

#12 Post by adrian_broher »

dbenage-cx wrote:Would you want a fresh install perspective now, or wait till issues are sorted with old sdk?
Thanks for the offer, but for now it's only pinning the old SDK design.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: CMakeify the SDK build process.

#13 Post by Vezzra »

adrian_broher wrote:I updated the README anyway. Could you please review it for completeness and comprehensibility?
Will try to get to it in the next days.
Also I set up a `mac-support` branch for this whole task in the repository.
Good idea.
And I think the issue reporting is probably done best on the repo issue tracker (https://github.com/freeorion/freeorion-sdk/issues).
Ok, done. I encountered the next issue, reported it there.

User avatar
Vezzra
Release Manager, Design
Posts: 6095
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: CMakeify the SDK build process.

#14 Post by Vezzra »

Ok, the build scripts do work now on OSX, up to the point where the zip archive is created. My question is, what to expect at that stage? An already fully functional SDK? Because that hasn't been achieved yet:
  • The zip archive only contains the contents of the produced "dep" folder, not the "dep" folder itself and the "bootstrap.command" file.
  • The structure of the produced "dep" folder is completely different from the "dep" folder in the current SDK.
  • In the current SDK, Ogg, Python, SDL2 and Vorbis are provided as OSX frameworks, those can probably be described as "library packages" which have a very specific internal structure. What your scripts produce are normal libraries, not OSX frameworks.
I didn't open a github issue because I don't know what to expect at the current stage... I guess recreating the structure of the "dep" folder of the current SDK is something you planned for a subsequent step?

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: CMakeify the SDK build process.

#15 Post by adrian_broher »

Vezzra wrote:My question is, what to expect at that stage? An already fully functional SDK?
In theory yes, there should be a deployable SDK at this point, some DMG file you just could take and upload to sourceforge without further editing.
The structure of the produced "dep" folder is completely different from the "dep" folder in the current SDK.
That was expected (from me) and as you already guessed that's the next part to fix. Also you can ignore the zip file for now it wasn't intended to build on anything else but windows So I need to make this step conditional. Instead of inspecting the zip file you can also look into the INSTALL directory, which is where all the dependencies where build into and packaged afterwards.
In the current SDK, Ogg, Python, SDL2 and Vorbis are provided as OSX frameworks, those can probably be described as "library packages" which have a very specific internal structure. What your scripts produce are normal libraries, not OSX frameworks.
CMake has some builtin support for creating first class OSX frameworks and I know that at least for the Ogg one it's just the matter of adding an additional switch. But I need to investigate on that a bit.
I didn't open a github issue because I don't know what to expect at the current stage... I guess recreating the structure of the "dep" folder of the current SDK is something you planned for a subsequent step?
Exactly. The best idea is probably to use the find tool here. When in the terminal you can navigate into the INSTALL directory and then calling something like

Code: Select all

find . | sort > ../cmake-sdk-layout.txt
This will contain enough information about where the current files are located. I can do the same here with the currently distributed FreeOrionSDK for MacOSX and compare those outputs. Just create an issue with the cmake-sdk-layout.txt uploaded/attached.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

Post Reply