Continous Integration Questions, Answers and Howto Thread

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Message
Author
User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Continous Integration Questions, Answers and Howto Thread

#1 Post by adrian_broher »

This thread should be a place you can ask questions or suggest new features for the Continuous Integration we use with FreeOrion. It isn't meant as a complete documentation but rather a rough outline of what CI is and can do for us.

But let start with a short overview of what is Continuous Integration, how it can help us making a better game and how the current infrastructure looks like:

What is Continuous Integration?

Continuous Integration (CI) is a term for the permanent and complete rebuild of a project from source in a clean build environment which is started after every source code change. In its most basic form this can be done by the developer himself before committing his code changes any may be limited to only rebuilding the application. However this is a very tedious task for the developer: Cleaning the build environment and rebuilding the whole application may be be complex and take some time, which can be better utilized for actual designing/development/bug hunting. Because of that the task of CI is usually done by a build server. The build server is some computer that does nothing else than listening to source code changes in a repository, rebuilding the application in a clean build environment and reporting the build status back to the developers.

CI isn't just limited to rebuilding the project and reporting compile errors. As long as you can write some test or check that can work without human intervention it can be used in CI. This can include, but is not limited to:
  • Unit testing (Does a small and isolated piece of code conform with some assertions written by developer)
  • Integration testing (Does a bigger and interacting piece of code conform with some assertions written by developer ;) )
  • Code Linting (Does the code follow a formatting style?)
  • Static code analysis (Is there some dubious code use, which may is a certain indicator for a hidden bug? E.g for C++: iterator use after erase, double free, null pointer dereferencing)
  • Code test coverage (Do the tests actually cover relevant pieces of application code, are there untested branches? If so, how much is not covered?)
  • Documentation coverage (Does documentation exist for a function, are all parameter documented? Are special code cases (e.g. Exception thrown) documented? Are the cross references valid?)
  • File asset format validation (Is the added texture a PNG file with 8 bit channels and either greyscale, RGB or RGBA?)
When adding all of those checks to the CI they are done for every single code change. This can aid the developer by pointing out flaws in his change early and helps code reviewers by doing tedious reviewing tasks so that the reviewer can focus on high level concepts like correct algorithm implementation/usage and code design rather than wrongly indented code.

On top of that CI may help with the software delivery (release). As a CI build is done from a clean build environment there it is way harder to create broken releases. Also a CI build may trigger not only on code changes, but on time events or code tagging too. This does allow regular releases on a scheduled time (nightly builds/weekly builds) or on tagging (release tags) without much human intervention.

What CI tools does FreeOrion use?

Currently we rely on several free services and tools to implement the CI. Those are:
  • GitHub - The place were we host the code on and which triggers via a webhook (HTTP requests send by GitHub on certain events) other services whenever a commit is done or a pull request is submitted.
  • TravisCI - A CI hoster, which provides virtual machines for the event of building source code. Travis provides Ubuntu and MacOSX based virtual machines to build the code and a web interface to both review the build reports and stop/rerun a build from the same commit. The configuration is done via a file called `.travis.yml` file in the root of the source repository, which is a conglomerate of configuration values and bash shell code snipplets.
  • AppVeyor - Another CI hoster, which does essentially the same as Travis, but provides virtual machines for Windows. The configuration is done via a file called `appveyor.yml` file in the root of the source repository, which is a conglomerate of configuration values and powershell code snipplets.
  • Docker - This is a virtualization software for applications, which does allows users to describe Linux virtual machine image by writing installation instructions for said image. It also consists of a hosting service to store those precomposed VM images and make them available for download. We use this tool to compose a virtual machine for Linux, which contains all build dependencies for FreeOrion. TravisCI by itself provides a rather slim and conservative designed virtual machine image of Ubuntu, which doesn't allow a somewhat recent C++11 build environment + all the special tools we would need for more complex CI tasks.
  • freeorion-bot - This is a GitHub/Docker account, solely created for the purpose of interacting with the services mentioned above. Currently when a build happens it is done on behalf of the freeorion-bot, not a certain developer. This will become more useful when weekly builds are a thing or then the CI will do semi-automatic code reviews.
What CI tasks are already in place for FreeOrion?

Currently we have the bare essentials in place.

For the `freeorion/freeorion` repository we do:
  • Build every commit/commit series and pull requests on Windows, Ubuntu 16.04 and MacOSX 10.10.
  • Report build issues on commits/commit series and pull requests on the GitHub web page and via developer eMail.
  • Rebuild C++ code documentation for every commit on `master` branch and upload it to http://freeorion.github.io/cppapi/master/
  • Build checks itself are limited to whether a build successfully compiles or not.

For the `freeorion/freeorion-sdk` repository we do:
  • Build every commit/commit series and pull requests on Windows and MacOSX 10.10, create a Docker image for `freeorion/freeorion` and update it if necessary.
  • Report build issues on commits/commit series and pull requests on the GitHub web page and via developer eMail.
  • Build checks itself are limited to whether a build successfully compiles or not.
  • Create release draft of the FreeOrionSDK from the build artifacts whenever a developer tags a release (in case you ever wondered why `freeorion-bot` is the one releasing the SDK and not me).
What are the limitations of the FreeOrion CI

As open source project we don't have any other investment than our interest, the knowledge and our time. This means we're pretty much limited to whatever we can get free of charge. All of GitHub, TravisCI and AppVeyor provides their services free of charge for open source projects but limit the use of said service. For example TravisCI limits the time to build to 50 minutes. If a build exceeds this time limit it is killed off. AppVeyor limits this to 60 minutes. When adding CI tools we need to keep this in mind. Also I would consider it a sign of good will to reduce the build time whenever possible, may it be by optimizing the code, the build or by other means and also keep in mind that we share these resources with other open source projects.

Another point are the capabilities of the CI in general. As already mentioned CI is limited to checks, which work non-interactive. When considering a new great tool for CI keep always in mind that the tool MUST work without a user typing commands into a shell or a user clicking away error messages because the virtual machines don't expose any user interface.

Also (but this is a personal interprecation of me) CI should never modify the project. That means the CI service should NEVER commit to the source or modify history. A CI should report issues, but not try to fix them. Maybe the CI does a dumb decision when trying to fix a bug, maybe the CI is buggy itself. The CI is a helper for tiresome tasks and a constant reminder for things that may cause problems, but the final decision is up to the developer.


If there are more questions regarding CI feel free to ask.

LGM-Doyle
Programmer
Posts: 219
Joined: Mon Feb 29, 2016 8:37 pm

Re: Continous Integration Questions, Answers and Howto Threa

#2 Post by LGM-Doyle »

adrian_broher, thanks for setting up the CI and explaining wit and offering to answer questions.

Thanks also for fixing the unit tests. They have been broken for a long time.

I have questions.

1. Do you know of specific techniques that improve build times?

2. Do you know how much the PIMPL idiom improves build times?

I use it to improve re-compilation times. I have not checked its impact on a clean build's duration.

3. You recently added a job to check PEP8 conformance. I've tried to reproduce freeorion's C++ style with some style checkers, clang and astyle. I was unsuccessful.

Could we tweak freeorion's C++ style so that it works with one of these style checkers? Adopting a style that can be checked with the CI would make that aspect of code reviews disappear.

If the answer to this question is yes, then it might deserve its own thread to answer what style checker we should use and what style conventions we are willing to change in order to facilitate automation.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Continous Integration Questions, Answers and Howto Threa

#3 Post by adrian_broher »

AAH!! I really need some warning like The post you're editing is not your own, continue?. Thankfully I still had the original tab open.
Thanks also for fixing the unit tests. They have been broken for a long time
Unfortunately the current state is still mostly broken and still needs repair and tweaking.
1. Do you know of specific techniques that improve build times?
In general I experienced the biggest improvements (in different projects unrelated to FO) when the amount of included headers was reduced. The problem with headers used is that included files may include 5 other files, which may in turn include around 2 to 10 files, which in turn … you get the idea. Suddenly you have thousands of file accesses during the compilation of a single file because someone used and include instead of a forward declaration. On a system with HDDs this can slow down the compile significantly. The other problem are templates. Whenever a template instantiated this happens in the context of the compilation unit. When using the sample template type ever compilation unit compiles the functions separately and the linker finally discard all duplicated instantiations.
2. Do you know how much the PIMPL idiom improves build times?

I use it to improve re-compilation times. I have not checked its impact on a clean build's duration.
Idoms like PIMPLs may help with the includes, but I don't feel like they should by used everywhere because they increase the code complexity in all dimensions (harder to understand, to write, a bit of runtime overhead. A more simple solution is the strict separation of independent classes into different headers. This cuts down the inclusion of unneeded headers significantly. Where meaning of independent may differ between cases. For example a lot of code need to know the declaration of ValueRef::ValueRefBase but only a small subset of code needs to know the declaration of subclasses of ValueRef::ValueRefBase. However the subclasses themselve have dependencies, which are completely irrelevant for most of the client code. Cutting down those dependencies should help a lot in terms of total and incremental compile time as code changes don't cascade through the code.

Regarding the templates the use of C++11 extern templates may help here. However I haven't collected enough experience with those yet.
3. You recently added a job to check PEP8 conformance.
No I haven't. The task is still open. The commits you're referring to are intended for IDE/editor use (and to gain experience with the tools). I'm still not sure how integrate this into the CI build. So far I'm reading up about the various external services like codeclimate or codecov and try to understand if they are a good fit into the current CI.
I've tried to reproduce freeorion's C++ style with some style checkers, clang and astyle. I was unsuccessful.
The code FO style conventions are IMO very special in the bad kind of way. There are various exceptions and conditions but lax rules in other places. I tried those tools too and failed the same way. They are not flexible enough to represent the formatting (which says enough about the formatting to be honest).
Could we tweak freeorion's C++ style so that it works with one of these style checkers? Adopting a style that can be checked with the CI would make that aspect of code reviews disappear.

If the answer to this question is yes, then it might deserve its own thread to answer what style checker we should use and what style conventions we are willing to change in order to facilitate automation.
Well, from the technical view it is of course it is possible. But is the team willing to change habits? Is there consensus on the bikeshed color? Also a bulk change of the existing code will cause a disruption in code history which I'm not very fond of. On the other hand reformatting code only when touched could lead to cases where the code will never change, because it works the way it exists.

In the end I consider the style enforcement not as important as running test, deployment and static code analysis which I would prefer to working on first.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

dbenage-cx
Programmer
Posts: 389
Joined: Sun Feb 14, 2016 12:08 am

Re: Continous Integration Questions, Answers and Howto Threa

#4 Post by dbenage-cx »

Any concerns with adding functionality to skip some builds?

For appveyor, looking at skipping for specific files:

Code: Select all

skip_commits:
  files:
    - default/scripting/
    - default/python/
    - default/data/art/**/*.jpg
    - default/data/sound/
    - default/data/fonts/
    - '**/*.txt'
    - '**/*.md'
    - '**/*.xml'
    - '**/COPYING'
For travis, at least looking at adding [skip travis], similar to integral [skip appveyor] (supposedly, I've not had a circumstance where it would have been useful yet to test).
Travis is a little more complex for file exclusion (e.g. facebook/react PR grepping git diff).
Any content posted should be considered licensed GNU GPL 2.0 and/or CC-BY-SA 3.0 as appropriate.

o01eg
Programmer
Posts: 2004
Joined: Sat Dec 10, 2011 5:46 am

Re: Continous Integration Questions, Answers and Howto Threa

#5 Post by o01eg »

I'm going to add long test for the game. What if split current BUILD_UNITTEST=true test on two separate jobs: one for UI and GG only and the second one with BUILD_HEADLESS=On enable for parser and game only?
Gentoo Linux x64, gcc-11.2, boost-1.78.0
Ubuntu Server 22.04 x64, gcc-12, boost-1.74.0
Welcome to the slow multiplayer game at freeorion-lt.dedyn.io.Version 2024-03-15.b3de094.
Donations're welcome:BTC:bc1q007qldm6eppqcukewtfkfcj0naut9njj7audnm

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Continous Integration Questions, Answers and Howto Threa

#6 Post by adrian_broher »

o01eg wrote:I'm going to add long test for the game. What if split current BUILD_UNITTEST=true test on two separate jobs: one for UI and GG only and the second one with BUILD_HEADLESS=On enable for parser and game only?
I don't like this idea, because the whole HEADLESS build is dubious for me. Separating by regular build (server+human client+ai client) and a unit test only build would be more logical to me. Also I need to investigate if we can share build artifacts between the stages so the test stage can use the artifacts generated by the build stage somehow.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Cjkjvfnby
AI Contributor
Posts: 539
Joined: Tue Jun 24, 2014 9:55 pm

Re: Continous Integration Questions, Answers and Howto Threa

#7 Post by Cjkjvfnby »

Test skipping:

Appveyor built ignore changes in specified locations: https://github.com/freeorion/freeorion/ ... or.yml#L12

If you want to skip builds for some reasons you can do it by adding one of the next entry to commit messages: All commits from the PR creation or from the last CI build execution should have an [must have such a skip] entry [in order for the test(s) to be skipped]. (edited by Dilvish for clarity)
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Continous Integration Questions, Answers and Howto Threa

#8 Post by Geoff the Medio »

Cjkjvfnby wrote:All commits from the PR creation or from the last CI build execution should have an [must have such a skip] entry [for the test(s) to be skipped]. (edited by Dilvish for clarity)
Could you edit that again / rephrase? It's not clear to me what this means.

Edit: Also, is it possible to disable parts of the CI build? eg. Something in the UI code changed, so I don't need a rebuild of the parsers, or in the server, so I don't need the client, AI, parsers, or GG code rebuilt.

User avatar
Dilvish
AI Lead and Programmer Emeritus
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Continous Integration Questions, Answers and Howto Threa

#9 Post by Dilvish »

Geoff the Medio wrote:
Cjkjvfnby wrote:All commits from the PR creation or from the last CI build execution should have an [must have such a skip] entry [for the test(s) to be skipped]. (edited by Dilvish for clarity)
Could you edit that again / rephrase? It's not clear to me what this means.
My understanding of the situation, which I had tried describing better, was that the CI checks would only be skipped if all of the new commits said to skip them. Would it clarify things more to say "in order for the tests to be skipped" rather than just "for the tests to be skipped"? (edit, I'll answer my own question and say 'yes, it would clarify more', and you can answer the implied question about whether that is clarified enough)
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Continous Integration Questions, Answers and Howto Threa

#10 Post by Geoff the Medio »

It seems to only run the test builds on the latest commit that was pushed, so I'd have thought marking that one would be enough.

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Continous Integration Questions, Answers and Howto Threa

#11 Post by adrian_broher »

Geoff the Medio wrote:Also, is it possible to disable parts of the CI build? eg. Something in the UI code changed, so I don't need a rebuild of the parsers, or in the server, so I don't need the client, AI, parsers, or GG code rebuilt.
Not quite sure what you're asking for.

TravisCI already does incremental builds for Linux and MacOS and ccache is smarter than the average developer on the decision what to and what not to compile bringing the compile times down to a few minutes total.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Cjkjvfnby
AI Contributor
Posts: 539
Joined: Tue Jun 24, 2014 9:55 pm

Re: Continous Integration Questions, Answers and Howto Thread

#12 Post by Cjkjvfnby »

Github released own CI. It is free for opensource.

Can replace both Travis and Appveyor.

https://github.com/features/actions
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Continous Integration Questions, Answers and Howto Thread

#13 Post by adrian_broher »

Cjkjvfnby wrote: Sun Apr 12, 2020 8:01 pm Github released own CI. It is free for opensource.

Can replace both Travis and Appveyor.

https://github.com/features/actions
Workflows run in Linux, macOS, Windows, and containers on GitHub-hosted machines, called 'runners'. Alternatively, you can also host your own runners to run workflows on machines you own or manage. For more information see, "About self-hosted runners."
Pretty beefy setup they provide there:

https://help.github.com/en/actions/gett ... age-limits
There are some limits on GitHub Actions usage, and will vary depending on whether you use GitHub-hosted or self-hosted runners. These limits are subject to change.

Job execution time - Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete. This limit does not apply to self-hosted runners.

Workflow run time - Each workflow run is limited to 72 hours. If a workflow run reaches this limit, the workflow run is cancelled. This limit also applies to self-hosted runners.

Job queue time - Each job for self-hosted runners can be queued for a maximum of 24 hours. If a self-hosted runner does not start executing the job within this limit, the job is terminated and fails to complete. This limit does not apply to GitHub-hosted runners.

API requests - You can execute up to 1000 API requests in an hour across all actions within a repository. If exceeded, additional API calls will fail, which might cause jobs to fail. This limit also applies to self-hosted runners.

Concurrent jobs - The number of concurrent jobs you can run in your account depends on your GitHub plan, as indicated in the following table. If exceeded, any additional jobs are queued. There are no concurrency limits for self-hosted runners.

GitHub plan Total concurrent jobs Maximum concurrent macOS jobs
Free 20 5
Pro 40 5
Team 60 5
Enterprise 180 50
Job matrix - A job matrix can generate a maximum of 256 jobs per workflow run. This limit also applies to self-hosted runners.
Certainly worth a look.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Cjkjvfnby
AI Contributor
Posts: 539
Joined: Tue Jun 24, 2014 9:55 pm

Re: Continous Integration Questions, Answers and Howto Thread

#14 Post by Cjkjvfnby »

I want to move python code style checks, python tests and string tables check to Github actions.

1) it is a separate runner, they will work in parallel with the existing ones.
2) it is possible to trigger them only when listed files/directories were changed

If not objections, I will test everything on my fork and provide instructions on how to set up it on the base repo.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
adrian_broher
Programmer
Posts: 1156
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Continous Integration Questions, Answers and Howto Thread

#15 Post by adrian_broher »

Cjkjvfnby wrote: Mon Jul 06, 2020 11:02 pm I want to move python code style checks, python tests and string tables check to Github actions.

1) it is a separate runner, they will work in parallel with the existing ones.
2) it is possible to trigger them only when listed files/directories were changed

If not objections, I will test everything on my fork and provide instructions on how to set up it on the base repo.
I already have a branch for this on my private repository with the primary purpose to test the Github actions itself, but it is still lacking the generation of code annotations from the pytest error report. Feel free to use it as a base for your effort:

https://github.com/freeorion/freeorion/ ... sts-action
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

Post Reply