extending Multi-Language-Support

Programmers discuss here anything related to FreeOrion programming. Primarily for the developers to discuss.

Moderator: Committer

Message
Author
User avatar
kroddn
Static Linker
Posts: 347
Joined: Thu Jun 28, 2007 10:28 am

extending Multi-Language-Support

#1 Post by kroddn »

Hello there,

today I worked on ger_stringtable.txt, because the game did not start any more when choosing this table.

So, I have some ideas for extending the code:
  • the API should do a failback and load strings from eng_stringtable.txt, if a text is not found in the translation. That avoids crashes. Maybe these strings could be marked by some special character ingame, and should be reported on the command line.
  • the translated language-files should be extended and contain the original english text, for example as a comment in the second line. That makes it easier for translaters to maintain the files and lets one easily find out, what has changed between two versions.
  • Even better would be, if the second line contains the english and the third line the translated text - if the third line is missing, then the english text could be choosen. This could be easily maintained by some shell-scripts, comparable to gettext()

tzlaine
Programming Lead Emeritus
Posts: 1092
Joined: Thu Jun 26, 2003 1:33 pm

Re: extending Multi-Language-Support

#2 Post by tzlaine »

kroddn wrote:Hello there,

today I worked on ger_stringtable.txt, because the game did not start any more when choosing this table.

So, I have some ideas for extending the code:
  • the API should do a failback and load strings from eng_stringtable.txt, if a text is not found in the translation. That avoids crashes. Maybe these strings could be marked by some special character ingame, and should be reported on the command line.
In-game, any missing entry is reported as "ERROR: X" where "X" is the string key. I haven't seen any crashes due to the stringtable. Could you report this as a bug on SF, with lots of detail, if you see this again?
[*] the translated language-files should be extended and contain the original english text, for example as a comment in the second line. That makes it easier for translaters to maintain the files and lets one easily find out, what has changed between two versions.
[*] Even better would be, if the second line contains the english and the third line the translated text - if the third line is missing, then the english text could be choosen. This could be easily maintained by some shell-scripts, comparable to gettext()[/list]
Patches welcome.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: extending Multi-Language-Support

#3 Post by Geoff the Medio »

The current crash appears to be due to a differing number of substitutions expected and available in strings. Various strings have %1% or %2% in them, which are replaced by some other text by boost::format. Unfortunately, boost::format throws exceptions when the number of such parameters differs from the number provided by the code.

This can be avoided by using the FlexibleFormat(...) function, defined in ClientUI.h, to do string substituion instead of boost::format any time a string does or might have substitution parameters, especially when then number of substitutions does or might change.

The problem comes up whenever new stringtable entries are added that expect substitutions to be available, but a corresponding string isn't added to the translation. The error text that indicates that the string is missing contains no substitution flags, so when substitutions are made without using FlexibleFormat, a crash results.

Also, there is an online translation system linked in the languages wiki template ( http://fo.lukaszk.net/ ) that displays the english and translated strings next to eachother. lukaszk doesn't seem to be spending any time working on it, however.

User avatar
kroddn
Static Linker
Posts: 347
Joined: Thu Jun 28, 2007 10:28 am

Re: extending Multi-Language-Support

#4 Post by kroddn »

Exactly that is the reason why I suggested these extensions. It would be easier to see changes if done this way.

@tzlaine: I already postet a bug-report, and also posted all information I got. There is NO error-message from boost.

/Edit: maybe there's a std::cerr somewhere which does not end with a linebreak and therefore the error-line is not printed. Adding "<< std::endl" to the end of those lines may be a good idea.

tzlaine
Programming Lead Emeritus
Posts: 1092
Joined: Thu Jun 26, 2003 1:33 pm

Re: extending Multi-Language-Support

#5 Post by tzlaine »

kroddn wrote:Exactly that is the reason why I suggested these extensions. It would be easier to see changes if done this way.
As always, patches are welcome.

User avatar
OndrejR
Space Dragon
Posts: 339
Joined: Thu Oct 02, 2008 11:00 pm
Location: Slovakia

Re: extending Multi-Language-Support

#6 Post by OndrejR »

kroddn wrote:Hello there,

today I worked on ger_stringtable.txt, because the game did not start any more when choosing this table.

So, I have some ideas for extending the code:
  • the API should do a failback and load strings from eng_stringtable.txt, if a text is not found in the translation. That avoids crashes. Maybe these strings could be marked by some special character ingame, and should be reported on the command line.
  • the translated language-files should be extended and contain the original english text, for example as a comment in the second line. That makes it easier for translaters to maintain the files and lets one easily find out, what has changed between two versions.
  • Even better would be, if the second line contains the english and the third line the translated text - if the third line is missing, then the english text could be choosen. This could be easily maintained by some shell-scripts, comparable to gettext()
I think that better idea than do it in game is to program convertor to do such thing. Currently in svn there are old translations(Spanish, Dutch, French) which are outdated. Before every release(or before someone want to translate) someone can run this convertor to fix translation automatically on all translations. If there is something missing or number of parameters don't match, then there will be added original english text. Also ordering will be same as in English text. And if some text is not needed anymore, then automatically it will be forgotten.

Later maybe there will be GUI for this or online site. Old site(FOOT) with translator is not working anymore and on this site they weren't considered updates from svn(or periodically integration with svn). If I sometime wanted to do such thing(I am from Slovakia and this language with Czech(which is related language) currently is unsupported in FreeOrion), then it have to be written in high level language such as Python or Java. Do it in C++ or in PHP is pain for me.

Problem with hosting is solvable with some freehosting(Python and Java), but question is if it shouldn't be more official.
Last edited by OndrejR on Tue May 26, 2009 3:39 am, edited 1 time in total.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: extending Multi-Language-Support

#7 Post by Geoff the Medio »

OndrejR wrote:I think that better idea than do it in game is to program convertor to do such thing. Currently in svn there are old translations(Spanish, Dutch, French) which are outdated. Before every release(or before someone want to translate) someone can run this convertor to fix translation automatically on all translations. If there is something missing or number of parameters don't match, then there will be added original english text. Also ordering will be same as in English text. And if some text is not needed anymore, then automatically it will be forgotten.
There are some issues with doing this automatically out of game.

Assume you have a program that reads the english stringtable and a translation, and replaces all the strings in the english table with the corresponding entries in the translation. The resulting mixed translation stringtable is written to a new updated translation file, which the game can load.

This works fine for adding strings or removing strings. Any new string added to english gets copied to the translation unchanged, and any removed string from english isn't substituted with the old translated version when running the program.

It also works for updating translations by editing the files to replace untranslated strings with translations. Any changes will be retained when running the program.

But how should changes to the english stringtable be handled if they're not additions or removal of strings? Pure text changes occur, in which a string is rewritten, and sometimes parameters in strings are added or removed.

If nothing special is done, the program that blindly replaces the english stringtable with entries from a translation will end up replacing newly changed english strings with old versions of english or the translation language that are in the translated stringtable, thinking it's copying a translated string over the different version in the english file.

The program could check changes between versions of the english stringtable. If an english string is changed in any way, the program could discard any translated version of that string and just output the english. This is potentially annoying for the translator, especially for trivial changes to the english file that don't change the meaning or any assumed formatting, but this would probably be safest.

Presumably some sort of marking of strings in the translation with comments could help, indicating that the string was translated but rejected due to a change in english, and what the rejected version of the translation was...

The program will need to preserved comments in the translation file, then, and copy them to the updated translation. But this is probably more difficult than it seems.

If comments from the translation are copied into the updated translation by the program, then it can't also copy comments from the original english translation. Otherwise, there'd be no way to tell which file a particular comment line came from, and whether it's already in the translation, or if it's changed, or if it's new and needs to be copied.

Strings also move around within the stringtable sometimes. Which comments should be associated with and moved with which strings? It could be the comments before a string in the english file or the translation, which would be placed before that string in its new location within the updated translation... This might make problems for section headings or similar comments, though, which would be harder to associated with a particular stringtable entry.
Later maybe there will be GUI for this or online site. Old site(FOOT) with translator is not working anymore and on this site they weren't considered updates from svn(or periodically integration with svn).
It's easy enough to get the latest version of the stringtable from SVN. There is web access, so any program with web access (regardless of whether it has a web interface) should be able to download the latest eng_stringtable.txt periodically or upon request.
If I sometime wanted to do such thing(I am from Slovakia and this language witch Czech(which is related language) currently is unsupported in FreeOrion), then it have to be written in high level language such as Python or Java. Do it in C++ or in PHP is pain for me.
A web interface would be ideal, as it greatly lowers the barrier to entry for translation contributions. Having to get python running and (I assume) running a command line program isn't that helpful. Java, if has to run in a little java applet box thing, wouldn't be very good either. Ideally the web-based translation tool would use actual web interface widgets and links to navigate.
Problem with hosting is solvable with some freehosting(Python and Java), but question is if it shouldn't be more official.
Any program written for translation should be hosted on freeorion.org, so we can maintain it. Disappearance of enthusiastic but temporarily volunteers offering hosting elsewhere makes them of limited use. The FOOT creator is such a case, despite the apparent utility of his tool, while it existed.

User avatar
OndrejR
Space Dragon
Posts: 339
Joined: Thu Oct 02, 2008 11:00 pm
Location: Slovakia

Re: extending Multi-Language-Support

#8 Post by OndrejR »

I did some research about how current open source projects do translation and there is already solution for this problem.

In FreeOrion there is translation file eng_stringtable.txt for English with

Code: Select all

# comment
PART_DESC_FIGHTER_STATS
'''Fighter Type: %1%
Fighter Anti-Fighter Damage: %2%
Fighter Anti-Ship Damage: %3%
Fighter Launch Rate: %4% fighters/turn
Fighter Speed: %6%
Fighter Health: %7%
Fighter Stealth: %8%
Fighter Detection: %9%
Capacity: %10% fighters'''
But as was described in previous posts there are a lot of problems with this and online translator will save precious developers time. And launchpad is such online translator for a lot of open source projects. For translation file there is standard named PO

Code: Select all

#: kxsldbgpart/libxsldbg/xsldbg.cpp:256
#, kde-format
msgid "%1 took %2 ms to complete."
msgstr "Trebalo je %2 ms da se %1 završi."
There is also a lot of GUI editors. More info about PO can be found here.

There are also supported templates named POT and there is also tools for merging. Online translator launchpad supports changed text and text needed for review.

statistics:
Image
There can be such thing that to untranslated languages can everybody post translation, but to partially translated only approved users and all translated text will be review before release.

There is also support for periodically import svn from external site to internal svn(if FO decides to stay in SourceForge) and also support for export.

During translation there are suggestions for that translation from all open source projects hosted on launchpad(including KDE, Ubuntu and a lot of others).
Image

There is one cons that all translations need to be in BSD license, but this license allows to change translations to LGPL or other appropriate license.

There are also some other props to not only change translations in FreeOrion to PO format and use launchpad site for translations, but also to move FreeOrion to launchpad itself. There is better bugs handling, supported code reviews(for example I propose change and somebody such as Geoff approves it or reject it until it is ok or until I get svn write access), better mailing list, support for answers(developer needs help) and also support for personal package archives(is comparison with SUSE factory?). There is also support for public releases(some bugs needed for next release to be solved)
Image

Image

There is also support for changelog and hash control sums and news. Launchpad can also import bugs from SourceForge and support mailing list.

I think that it is time for change to PO format for FreeOrion and to possibly also to move to launchpad. Here is Ubuntu launchpad site. SourceForge is outdated.

User avatar
OndrejR
Space Dragon
Posts: 339
Joined: Thu Oct 02, 2008 11:00 pm
Location: Slovakia

Re: extending Multi-Language-Support

#9 Post by OndrejR »

GUI editors for PO:
Poedit
Lokalize
gTranslator

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: extending Multi-Language-Support

#10 Post by Geoff the Medio »

I have a couple concerns.

Skimming over the pages you linked, the PO format doesn't seem well suited to our setup where strings are represented in code by a lookup identifier. We'd seem to need an english translation, that would contain mappings from all the CODE_STRING_NAMES to human-readable english "Code String Names". But then how would that work for translations? The file being translated would just contain msgid "CODE_STRING_NAMES" which doesn't help the translator much. Is there a way to easily support a system that already uses lookup string names in a stringtable, rather than the gettext system PO seems to be designed for?

Also, all FreeOrion in-game text is CC-BY-SA licensed. I don't think CC-BY-SA content can be relicensed under a BSD license, and a translation is a derivative work as far as I know, so would need to be CC-BY-SA as well, due to the SA = Share Alike.

User avatar
OndrejR
Space Dragon
Posts: 339
Joined: Thu Oct 02, 2008 11:00 pm
Location: Slovakia

Re: extending Multi-Language-Support

#11 Post by OndrejR »

Geoff the Medio wrote:Skimming over the pages you linked, the PO format doesn't seem well suited to our setup where strings are represented in code by a lookup identifier. We'd seem to need an English translation, that would contain mappings from all the CODE_STRING_NAMES to human-readable English "Code String Names". But then how would that work for translations? The file being translated would just contain msgid "CODE_STRING_NAMES" which doesn't help the translator much. Is there a way to easily support a system that already uses look up string names in a stringtable, rather than the gettext system PO seems to be designed for?
There are more possibilities(theoretically described in link which I posted about PO in section "Enter The PO Format"):
  • replace all CODE_STRING_NAMES with "Code String Names" in FreeOrion and use standard gettext. But changing English string "Fighters" to "Fighters %1" means also replacing all occurences of this text in FO code.
  • Use current English string table with CODE_STRING_NAMES in FreeOrion code. But this English string table will be transformed into English PO with msgid "Code String Names", msgstring "Code String Names" and msgctxt "CODE_STRING_NAMES". There have to be condition of unique msgid, but this is solvable during string table transformation process to PO. Before release there will be export from launchpad to other languages translation PO(English PO will be periodically created in FreeOrion svn) and this other language PO can be transformed back to other language string table.
  • Same as previous with one modification - there will be no transformation back from other language PO to string table. Instead UserString will be modified to load string table english text and then will use it as parameter to gettext function.
I think that third option is best, but first to implement is better second and then in next iteration implement also third option.
Geoff the Medio wrote:Also, all FreeOrion in-game text is CC-BY-SA licensed. I don't think CC-BY-SA content can be relicensed under a BSD license, and a translation is a derivative work as far as I know, so would need to be CC-BY-SA as well, due to the SA = Share Alike.
launchpad wrote:I have no problem with BSD myself, but I also uploaded translations from upstream. What do I do?

As long as the uploads were marked as translations that were published elsewhere, they fall under a separate copyright regime: those imports will retain their original copyright license. The licensing cleanup only applies to translations that are (as far as the system knows) original to Launchpad. .
As far as I understand that, only new translations are under BSD license although I don't know how it works in practice. But we can always ask translator for dual license BSD and CC-BY-SA for existing translations. And all new translations will be done on launchpad with BSD license and after exporting this translations to FreeOrion they would have CC-BY-SA after changing license from BSD(BSD allows that). English text is not translation and therefore it has CC-BY-SA.

To the existing translations - there will be needed to transform them to PO to import them to launchpad(and review) and possibly one time change license.

User avatar
OndrejR
Space Dragon
Posts: 339
Joined: Thu Oct 02, 2008 11:00 pm
Location: Slovakia

Re: extending Multi-Language-Support

#12 Post by OndrejR »

There is another online translator with standardized PO format and is developed at SourceForge. Has name Pootle. It is translator server which FreeOrion may host on its site freeorion.org. There is possibility to create translator user which can commit translations to svn and in Pootle users with given rights to commit can use Pootle mechanism to commit translation to svn.

There is no need for BSD license and according Pootle features it has also suggestions, reviews, assigning tasks, access rights, GUI tools. Here is Pootle User Guide created by OpenOffice.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: extending Multi-Language-Support

#13 Post by Geoff the Medio »

OndrejR wrote:As far as I understand that, only new translations are under BSD license although I don't know how it works in practice. But we can always ask translator for dual license BSD and CC-BY-SA for existing translations.
I don't think we can ask that... Although the launchpad site seems to think it's possible to put in original text in some other license and have the translations be released under a/the BSD license, the CC-BY-SA says,
CC-BY-SA 3.0 wrote:"Adaptation" means a work based upon the Work, or upon the Work and other pre-existing works, such as a translation...
which, with other parts of the license, that a translation of CC-BY-SA text must only be released under a CC-BY-SA compatible license.

I searched a bit but can't easily find out if there is a BSD license that's CC-BY-SA 3.0 compatible. BSD is a software license, so it's not really clear how it works for text and translations. I can't find any direct comparisons (as opposed to discussions about BSD code being released under GPL or CC-BY-SA vs. GFDL) to clarify.

User avatar
OndrejR
Space Dragon
Posts: 339
Joined: Thu Oct 02, 2008 11:00 pm
Location: Slovakia

Re: extending Multi-Language-Support

#14 Post by OndrejR »

Geoff the Medio wrote:
OndrejR wrote:As far as I understand that, only new translations are under BSD license although I don't know how it works in practice. But we can always ask translator for dual license BSD and CC-BY-SA for existing translations.
I don't think we can ask that... Although the launchpad site seems to think it's possible to put in original text in some other license and have the translations be released under a/the BSD license, the CC-BY-SA says,
CC-BY-SA 3.0 wrote:"Adaptation" means a work based upon the Work, or upon the Work and other pre-existing works, such as a translation...
which, with other parts of the license, that a translation of CC-BY-SA text must only be released under a CC-BY-SA compatible license.
I thought to ask people who translated text - email them and ask to release their translation also in BSD license. Not to change existing CC-BY-SA license to BSD. Authors have rights to release their work also in another license.

If it will be not required to do this, then existing translations in svn have CC-BY-SA license, in launchpad is the same and only new translations will have on launchpad BSD license.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 13587
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: extending Multi-Language-Support

#15 Post by Geoff the Medio »

OndrejR wrote:I thought to ask people who translated text - email them and ask to release their translation also in BSD license. Not to change existing CC-BY-SA license to BSD. Authors have rights to release their work also in another license.

If it will be not required to do this, then existing translations in svn have CC-BY-SA license, in launchpad is the same and only new translations will have on launchpad BSD license
Translations of CC-BY-SA works are not new works; they are adaptations of existing work. The CC-BY-SA specifically says this.

Consequently, if an existing work is licensed under CC-BY-SA, adaptation of the existing work can only be released under compatible licenses.

"Authors" of adapted works have no rights to rerelease a work beyond those rights granted by the license of the original work.

This is what's confusing to me about the launchpad system. I don't see how it works for original text, unless that text is released under a BSD-compatible license.

Post Reply