Build number issue

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderators: Committer, Committer

Message
Author
User avatar
Vezzra
Release Manager, Design
Posts: 5004
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Build number issue

#1 Post by Vezzra » Wed Jul 15, 2015 7:07 pm

In commit 3323243 this:

Code: Select all

    commit = check_output(["git", "show", "-s", "--format=%h", "HEAD"]).strip()
    timestamp = float(check_output(["git", "show", "-s", "--format=%ct", "HEAD"]).strip())
    build_no = ".".join([datetime.utcfromtimestamp(timestamp).strftime("%Y-%m-%d"), commit])
...has been simplified to:

Code: Select all

    build_no = check_output(['git', 'show', '-s', '--format=%cd.%h', '--date=short', 'HEAD'], shell=False).strip()
However, with that change the conversion of the timestamp to UTC has been removed.

My first concern was that the "git show" command might convert the committer date to local time, which of course would have resulted in possible different build numbers, depending on the time zone of the computer on which a build is produced. However, this isn't the case, as the "git show" command returns the timestamp in the local time of the original committer. Which in turn creates another problem: If, for example, Dilvish (who is at -0700) makes a commit 2 hours after me (I'm at +0200, so 9 hours ahead), his commit, although the later one, has an "earlier" timestamp. Resulting in a possibly "lower" build number for the later commit.

That of course defeats the whole point of composing our build number the way it's done now, which is to have consecutive, incremental build numbers. Therefore, as a quick solution, I suggest to revert back to the original implementation which converts the timestamp to UTC before using it for the build number. This should take care of the issue.

I've created a PR with the respective changes.

But this issue illustrates quite nicely that using a date as build number isn't the most brilliant idea I ever had (Marcel and I had a brief exchange about that via PM). So I'm once again looking for a better way to compose our build number. Suggestions welcome, the best I've come up so far is the following:

Tag the very first commit of the repo with an annotated tag like "build-no-base" and simply use "git describe --match build-no-base" to get the total number of commits. Use this as the first part of the build number, and the hash as the second part (like it's already done now).

Comments?

User avatar
Vezzra
Release Manager, Design
Posts: 5004
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build number issue

#2 Post by Vezzra » Fri Jul 24, 2015 3:07 pm

Continuing a discussion started here, but fits better here in this thread:
Chriss wrote:The "what version number to use" question is probably not specific to arch...
Yep, we've already had issues with our current solution for composing the build number, see above. The "git rev-list --count HEAD" command in your OP in this thread although would accomplish the same as my suggestion to tag the first commit with "build-no-base" and use "git describe --match build-no-base" to get the total number of commits, but without the need for the tag.

That looks far better (and not to mention, more compatible with FO packages for Linux distros) than what we have now, but so far I've got no comments on my suggestion. The change can be made quite quickly (still in time for 0.4.5), but this is not my decision alone...

Geoff, Dilvish, Mat, Marcel?

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Build number issue

#3 Post by adrian_broher » Fri Jul 24, 2015 3:20 pm

Vezzra wrote: […] but so far I've got no comments on my suggestion
[…] Marcel?
Still don't care and still see the git hash only as sufficient.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
MatGB
Creative Contributor
Posts: 3310
Joined: Fri Jun 28, 2013 11:45 pm

Re: Build number issue

#4 Post by MatGB » Fri Jul 24, 2015 3:57 pm

It's one of those "above my paygrade" things, I like having the date in it, but always managed perfectly with the SVN number, the hash isn't as clear as a rising number but if that's easiest it's fine, at least for the test builds, probably best to have some sort of date in the release builds but not totally necessary.
Mat Bowles

Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Build number issue

#5 Post by Dilvish » Fri Jul 24, 2015 6:03 pm

I seem to have a slight preference for the date over the simple commit count, but it seems a pure personal preference (and a minor one at that) with on particular supporting reason I can identify.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Build number issue

#6 Post by Dilvish » Fri Jul 24, 2015 6:19 pm

Thinking a bit more on the topic, I should add-- my thinking and comment thus far had really been more around the build number to use in our regular weekly builds, not for the 'Stable Release'. For the latter I could certainly understand wanting to drop the date.

On the total commit count thing, though, I am concerned it is not really monotonically increasing like I have the impression you consider it to be. I believe that Git garbage collection can cause the number to suddenly and unexpectedly drop, and of course it would also vary tremendously by person depending on how many PRs you had downloaded or other experimental branches you had.
Eventually, Git will decide that it's time to run garbage collection. (You can trigger this process yourself, using git gc.) Starting from every branch and every tag, Git walks back through the graph, building a list of every commit it can reach. Once it's reached the end of every path, it deletes all the commits it didn't visit.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Build number issue

#7 Post by adrian_broher » Fri Jul 24, 2015 6:44 pm

Dilvish wrote:I believe that Git garbage collection can cause the number to suddenly and unexpectedly drop, and of course it would also vary tremendously by person depending on how many PRs you had downloaded or other experimental branches you had.
I don't follow you how you came to this conclusion that git rev-list does not return a monotonic increasing number.

quoting the manual:
git-rev-list wrote:List commits that are reachable by following the parent links from the given commit(s)
and
git-gc wrote: Runs a number of housekeeping tasks within the current repository, such as compressing file revisions (to reduce disk space and increase performance) and removing unreachable objects which may have been created from prior invocations of git add.
TL;DR:
git-rev-list only counts reachable commits.
The Git GC deletes only unreachable commits.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Build number issue

#8 Post by Dilvish » Fri Jul 24, 2015 8:32 pm

adrian_broher wrote:git-rev-list only counts reachable commits.
The Git GC deletes only unreachable commits.
From my reading, unless you add --all (or perhaps --branches?) to your rev-list command, then these lines would not actually talking about the same partition of reachable/unreachable". If you do add such an option, then this summary would support my point, since adding and deleting branches and whatnot is exactly the kind of user action I referred to as changing the total commit count.

Furthermore, if an explicit reference to "git-rev-list" was in this thread before this, I hadn't noticed it. I was responding to
and simply use "git describe --match build-no-base" to get the total number of commits
Even looking into git describe a little I didn't immediately see how that was going to give a number of commits, so I just focused on "the total number of commits"
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Build number issue

#9 Post by adrian_broher » Fri Jul 24, 2015 8:44 pm

Dilvish wrote:From my reading, unless you add --all (or perhaps --branches?) to your rev-list command, then these lines would not actually talking about the same partition of reachable/unreachable". If you do add such an option, then this summary would support my point, since adding and deleting branches and whatnot is exactly the kind of user action I referred to as changing the total commit count.
Of course you don't want to count commits on branches and only want to count parent/ancestor commits relative to the commit you declare as starting point (HEAD in this case) to the initial commit.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Build number issue

#10 Post by Chriss » Fri Jul 24, 2015 9:56 pm

adrian_broher wrote:
Vezzra wrote: […] but so far I've got no comments on my suggestion
[…] Marcel?
Still don't care and still see the git hash only as sufficient.
To describe the code someone refers to, the hash is perfect. BUT, for packaging, I need to have a version which can be ordered. I (or rather the package manager) need to know which package is newer, so I know when to update the package. The Git hash has no order. It is useless for that use case. For packaging, it is NOT sufficient.
Dilvish wrote:
adrian_broher wrote:git-rev-list only counts reachable commits.
The Git GC deletes only unreachable commits.
From my reading, unless you add --all (or perhaps --branches?) to your rev-list command, then these lines would not actually talking about the same partition of reachable/unreachable". If you do add such an option, then this summary would support my point, since adding and deleting branches and whatnot is exactly the kind of user action I referred to as changing the total commit count.
As far as I understand this: By specifying a commit, you specify a branch and everything else. You refer to one specific state of the code. Yes, your total commit count in your local repository may vary, but then you're on a different branch, and / or have more commits, so you don't refer to the same state of the code.

You don't even really want to count all commits in your repo. All you really need to count is the number of commits it took to reach that specific state of the code. That count can even restart if it is prefixed by a version tag like one of my examples does, and that version tag increments in a way that the package manager understands. So maybe a Build number could be "0.4.4+r<commit-count>.<git hash>"? Not sure if the "+" works, as in: the relevant package managers understand that 0.4.4 < 0.4.4+ < 0.4.5.
Attached patches are released under GPL 2.0 or later.

User avatar
Dilvish
AI Lead, Programmer
Posts: 4768
Joined: Sat Sep 22, 2012 6:25 pm

Re: Build number issue

#11 Post by Dilvish » Fri Jul 24, 2015 10:26 pm

Chriss wrote:So maybe a Build number could be "0.4.4+r<commit-count>.<git hash>"? Not sure if the "+" works, as in: the relevant package managers understand that 0.4.4 < 0.4.4+ < 0.4.5.
How about simply "0.4.4.<commit-count>.<git hash>"? (and dropping the ".<git hash>" for the stable releases.) If we have the commit count in there I see no need anymore for the "+", nor the "r" really either.
If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12459
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Build number issue

#12 Post by Geoff the Medio » Sat Jul 25, 2015 7:11 am

Dilvish wrote:...I see no need anymore for the "+", nor the "r" really either.
+ is still useful, to indicate that it's after the previously-indicated release, without requiring people to know what commit id or count corresponds to what time point / release or compare the numbers.

Chriss
Dyson Forest
Posts: 231
Joined: Sun May 11, 2008 10:50 am

Re: Build number issue

#13 Post by Chriss » Sat Jul 25, 2015 2:16 pm

AFAIK, the r is something which arch uses so the revision can be recognized better. I can add that if you want to drop it.
Attached patches are released under GPL 2.0 or later.

User avatar
Vezzra
Release Manager, Design
Posts: 5004
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Build number issue

#14 Post by Vezzra » Sun Jul 26, 2015 3:44 pm

Dilvish wrote:I just focused on "the total number of commits"
Sorry, unclear wording on my part. What I meant was that:
adrian_broher wrote:Of course you don't want to count commits on branches and only want to count parent/ancestor commits relative to the commit you declare as starting point (HEAD in this case) to the initial commit.
Which is, as far as I understand, exactly what "git rev-list --count <commit-ref>" does. So the output of that command should yield a stable result independent on what branches that do not lead to the referenced commit the local repo contains. Meaning, it should produce the same result regardless of who builds a certain commit, as the parent/ancestor commit history should be the same.

Which should serve our purpose just fine.

User avatar
adrian_broher
Programmer
Posts: 1082
Joined: Fri Mar 01, 2013 9:52 am
Location: Germany

Re: Build number issue

#15 Post by adrian_broher » Sun Jul 26, 2015 4:05 pm

Vezzra wrote:
adrian_broher wrote:Of course you don't want to count commits on branches and only want to count parent/ancestor commits relative to the commit you declare as starting point (HEAD in this case) to the initial commit.
Which is, as far as I understand, exactly what "git rev-list --count <commit-ref>" does.
You understood that correctly.
Resident code gremlin
Attached patches are released under GPL 2.0 or later.
Git author: Marcel Metz

Post Reply