Translations

[XuaXua]

If you export something via Transifex/Launchpad/etc, it almost always gives you a Unicode tex... hm, yes, Xua, you are right!

I have over a decade of I18N experience under my belt. The worst is working for companies who use the default database encoding.

[mumra]

If you export something via Transifex/Launchpad/etc, it almost always gives you a Unicode tex... hm, yes, Xua, you are right!

I have over a decade of I18N experience under my belt. The worst is working for companies who use the default database encoding.

That's a horrible problem I've run into before on mysql. I can't remember the exact context; basically there was a database with Swedish Latin (which is mysql's slightly bizarre default setting), but some tables had been created with UTF8, or something like that. Was a right mess sorting it all out!

[XuaXua]

If you export something via Transifex/Launchpad/etc, it almost always gives you a Unicode tex... hm, yes, Xua, you are right!

I have over a decade of I18N experience under my belt. The worst is working for companies who use the default database encoding.

That's a horrible problem I've run into before on mysql. I can't remember the exact context; basically there was a database with Swedish Latin (which is mysql's slightly bizarre default setting), but some tables had been created with UTF8, or something like that. Was a right mess sorting it all out!

Since the database involved here is likely flat text files (akin to Java's "Resource Bundles"), the other primary issue I've encountered concerning multiple developers on the same I18N project (the Stone Soup, if you will, of editor applications being used to edit a text, seemingly with the proper encoding set) comes into play. I've fallen in the bad editor trap twice; once when I first discovered it, and another time when another developer took on I18N responsibility of his own accord, did a find and replace, and ruined a deployment. Rollback was easy, but consistency, rule enforcement, and training are keys to prevention.

[MyOtheHedgeFox]

I think focusing on these nifties is just losing time now...

[Grimm]

I've set up a Transifex page for us here https://www.transifex.net/projects/p/dcss/

Interested translators please sign up and take a look. Let's evaluate it and decide whether it's what we need.

[dk]

I've set up a Transifex page for us here https://www.transifex.net/projects/p/dcss/

Interested translators please sign up and take a look. Let's evaluate it and decide whether it's what we need.

I'm not 100% sure how the workflow is/should be with Transifex, but would you mind invite me to the project (my username is bigyellowcactus)?

[TehDruid]

You have my sword... Ok. I'm in, now I have to see how this works. How should I start translating, first of all? Guess I have to get the source code first of all?

[MyOtheHedgeFox]

If you translate a game, you should have:
1) Something you translate;
2) A tool to convert (1) into game-usable format;
3) A way to see how it looks in the game (source code, a special tool, daily "please make a new test version" mail to the game developers...).


If you are going to translate the game, GET THE SOURCE CODE.
You must be able to find which each string is used.

<A lot of blabber was cut out>

[TehDruid]

I've compiled the game in Win7 some months ago cause I was trying to add some unrandarts of my own, no worries. I'll just have to freshen my memory.

So, basically, I find the appropriate piece of text in the source files, change it and then compile the game to see how it plays out. Perhaps this is a stupid question, but can't we translate without having to compile the game every time we want to check if our work appears correctly? That's going to take quite a while.

I'll give it a go after I'm back from class tomorrow morning, anyway.

[MyOtheHedgeFox]

Erm... Saw my post above? That was a nice example of Yiuf-class nonsense caused by overstudy.

I beg your pardon for all the nonsense I wrote in the previous message. I do not think it will work right now, so what you should do at the moment is translate all the strings using Grimm's Transifex Workshop.

[galehar]

I've compiled the game in Win7 some months ago cause I was trying to add some unrandarts of my own, no worries. I'll just have to freshen my memory.

So, basically, I find the appropriate piece of text in the source files, change it and then compile the game to see how it plays out. Perhaps this is a stupid question, but can't we translate without having to compile the game every time we want to check if our work appears correctly? That's going to take quite a while.

I'll give it a go after I'm back from class tomorrow morning, anyway.

If you want to test locally a change in descriptions or database, there is no need for compiling. Just edit the file in dat/descript or dat/database and restart the game. But you should submit your translations or improved english descriptions on transiflex once the project is properly set up.

[TehDruid]

@MyO: Very well. I'll have to converse with Grimm then. Btw I didn't find your previous post annoying or anything.

@Galehar: Thank you!

[jpeg]

How should I start translating, first of all? Guess I have to get the source code first of all?

Seeing how it's currently just descriptions that are being translated, you shouldn't require the source code at all. If you need to check how your translation looks in-game (which might be important to see how much fits the screen in a 24x80 window or whether special characters are getting displayed correctly) you can directly copy/paste your translation into the corresponding description file (in the dat/descript/ directory).

Later on, when dynamic messages (even stuff like speech) get translated, checking the results in-game will become more important, but even then compiling shouldn't be necessary, seeing how all messages are being outsourced to the dat/ directory.

EDIT: That's what I get for taking so long to compose my posts, and getting interrupted by actual work...

[CommanderC]

Right now the main obstacle I see is that we need to import all those files in dat/descript into transifex. I don't know about the status of galehar's script, but I have made this (ugly) python script that maybe can be handy. It transforms the .txt files into .yaml files. It should be run in the root directory of crawl (the one that contains source, docs, etc)

  Code:

#!/usr/bin/env python
import os
import os.path
import yaml

def parse(input_file):
    with open(input_file) as f:
        state = 0
        key = ""
        value = []
        for l in f:
            if l[0] == '#' or len(l) == 1:
                pass
            elif l[0:4] == '%%%%':
                if state == 1:
                    yield key, ''.join(value)
                state = 2
            elif state == 2:
                key = l
                value = []
                state = 1
            elif state == 1:
                value.append(l)
               
def process(input_file, output_file, locale):
    with open(output_file, 'w') as f:
        data = {}
        for k,v in parse(input_file):
            data[k] = v
        result = {locale:data}
        yaml.dump(result, f)

def main():
    output_dir = 'output'
    input_dir = os.path.join('source', 'dat', 'descript')
    files = ['ability', 'branches', 'commands', 'gods', 'monsters', 'skills', 'spells', 'unrand', 'backgrounds', 'cards', 'features', 'items', 'species', 'unident']
    all_locales = ['de', 'es', 'ru']
    for locale in all_locales:
        for f in files:
            output_dir2 = os.path.join(output_dir, locale)
            if not os.path.exists(output_dir2):
                os.makedirs(output_dir2)
            output_file = os.path.join(output_dir, locale, f + '.yaml')
            input_file = os.path.join(input_dir, f + '.txt')
            process(input_file, output_file, locale)

if __name__ == '__main__':
    main()



[galehar]

Here it is. Not much prettier than yours
Should be easy to adapt it to make it spit yaml output instead of dokuwiki.
Right now all our scripts in util are written in perl, and kilobyte (who is doing translations) use perl for scripting. So if the export script were to be commited in trunk, it would be better if it were written in perl.
Python is fine too, if you guys polish a python script and maintain it, we won't turn it down.

  Code:

#!/usr/bin/perl

use strict;
use warnings;
use Text::Wrap;
$Text::Wrap::columns = 80;
$Text::Wrap::separator = " \\\\ ";

my $base_dir = `git rev-parse --show-toplevel`;
chomp $base_dir;
#my $quotes =  "$base_dir/crawl-ref/source/dat/database/quotes.txt";
#die "Cannot open $quotes for reading\n" unless (-r $quotes);
die "Usage: $0 description_files\n" unless (@ARGV);

my %DESCRIPTIONS;
#my %QUOTES;

main();

sub main {
#   load_file($quotes, \%QUOTES);
   foreach (@ARGV) {
      load_file($_, \%DESCRIPTIONS);
   }
   gen_array();
}

sub load_file {
   my ($input, $hash) = @_;
   open IN, $input or die "Cannot open $input";
   my ($key, $value);
   while(<IN>) {
      next if (/^#/ or /^\s*$/);
      chomp;
      if (/^%%%%$/) {
         if ($key) {
            $value =~ s/([{}])/\\$1/g;
            $hash->{$key} = wrap("", "", $value);
         }
         $key = "";
         $value = "";
         next;
      }
      if ($key) {
         $value .= $_;
         $value .= " \\\\ " if /^\s+/;
      } else {
         $key = $_;
      }
   }
   close IN;
}

sub gen_array {
   print "^Original^^Translation^^ ^ ^\n";
   print "^Name^Description^Name^Description^Review^Notes^\n";
   foreach (sort keys %DESCRIPTIONS) {
      printf "|%s|%s| | | |\n", $_, $DESCRIPTIONS{$_};
   }
}


[CommanderC]

Thanks, galehar. I have modified your script. Here it is:

  Code:

#!/usr/bin/perl

use strict;
use warnings;
use YAML ();
#local $YAML::SortKeys = 0;
#use Text::Wrap;
#$Text::Wrap::columns = 80;
#$Text::Wrap::separator = " \\\\ ";

my $base_dir = `git rev-parse --show-toplevel`;
chomp $base_dir;
#my $quotes =  "$base_dir/crawl-ref/source/dat/database/quotes.txt";
#die "Cannot open $quotes for reading\n" unless (-r $quotes);
die "Usage: $0 description_files\n" unless (@ARGV);

my %DESCRIPTIONS;
#my %QUOTES;

main();

sub main {
#   load_file($quotes, \%QUOTES);
   foreach (@ARGV) {
      load_file($_, \%DESCRIPTIONS);
   }
   gen_array();
}

sub load_file {
   my ($input, $hash) = @_;
   open IN, $input or die "Cannot open $input";
   my ($key, $value);
   while(<IN>) {
      next if (/^#/ or /^\s*$/);
      chomp;
      if (/^%%%%$/) {
         if ($key) {
            $value =~ s/([{}])/\\$1/g;
#            $hash->{$key} = wrap("", "", $value);
            $hash->{$key} = $value;
         }
         $key = "";
         $value = "";
         next;
      }
      if ($key) {
         $value .= $_;
         $value .= " \\\\ " if /^\s+/;
      } else {
         $key = $_;
      }
   }
   close IN;
}

sub gen_array {
   my $data = [];
   my $i = 0;
   foreach (sort keys %DESCRIPTIONS) {
       $data->[$i++] = {
           "Original Name" => $_,
           "Original Description" => $DESCRIPTIONS{$_},
           "Translated Name" => "",
           "Translated Description" => "",
           "Review" => "",
           "Notes" => ""
       };
   }   
   print YAML::Dump({en => $data});
}



[Grimm]

Let's slow down a moment and make sure Transifex is going to work. I have no experience with this sort of program. I am currently unable to figure out how to import text into Transifex - the documentation says that we can input a gitorious link to transfer text but I can't find the page that allows that. I also don't see how I can give people "maintainer" status.

Also, if you can use your Crawl name there that would help us keep clear who is who.

[MyOtheHedgeFox]

Once you created a language group, the one who requested its creation automatically becomes its maintainer.

[Grimm]

I've entered CREDITS.txt from 0.10.1 as a resource on Transifex, let's try translating that to see how it goes.

Now I see where the URL can go. Is http://gitorious.org/crawl/crawl/blobs/ ... REDITS.txt
the location of the 0.10.1 version? That's what I've linked to at the moment.

[Kate]

That's master. For 0.10, you want http://gitorious.org/crawl/crawl/blobs/ ... REDITS.txt

[dk]

German version of the credits added! Yeah!

OK, took only some minutes...

FYI: If working with plaintext files, the translated file has to have the same number of lines as the original. That sucks

[dk]

OK, it seems the Changelog.txt-file changed. Now there's a html-page from gitorious instead of the actual text-file and my translations is lost...

[dk]

That's master. For 0.10, you want http://gitorious.org/crawl/crawl/blobs/ ... REDITS.txt

I think for Transifex we have to use http://gitorious.org/crawl/crawl/blobs/raw/stone_soup-0.10/crawl-ref/CREDITS.txt instead of http://gitorious.org/crawl/crawl/blobs/stone_soup-0.10/crawl-ref/CREDITS.txt, since we don't want to translate the html-page

[TehDruid]

I opened the current file that is on Transifex on Notepad++ and it contains html code instead of just text. So I'm kinda clueless on what to translate and what to not translate of it. Any advice would be appreciated.

[dk]

I opened the current file that is on Transifex on Notepad++ and it contains html code instead of just text. So I'm kinda clueless on what to translate and what to not translate of it. Any advice would be appreciated.

It seems to be a problem with the url of the file. Transifex seems to check out the file automatically, but the url links to the gitorious page instead of the actual file

[TehDruid]

Should I wait for that to be corrected then, or download 0.10 (or is it 0.10.1 we're working on? Is there a difference in text between those?) and translate it locally then upload it to Transifex?

[dk]

You can download it here: http://gitorious.org/crawl/crawl/blobs/ ... REDITS.txt

But before uploading, we have to wait 'till this issue is fixed.

[Grimm]

I've changed it to http://gitorious.org/crawl/crawl/blobs/ ... REDITS.txt but now it shows all the html. Perhaps it would be simpler to upload the .txt files by hand.

[dk]

Maybe it is because Transifex did not pull the file directly after you changed the url. Is there a way to force Transifex to pull the file from git?
This morning I downloaded the credits.txt with no problems, uploaded my translation, and some minutes later I wondered why my translation was lost. I noticed the the time that is displayed right next to the resource file changed to Mar 30, 08:30a.m. This seems to be the point in time where Transifex synchronized the files and downloaded the html-page, which then overwrote the real credits.txt file uploaded by hand earlier.

[MyOtheHedgeFox]

Is there a way to access the git repo as if it was an ftp server?

[dk]

you cann access the git repo via http/https: http://gitorious.org/crawl

[TehDruid]

I'm done with CREDITS.TXT although I had to make it some lines longer because I couldn't keep within the limits of each line. Oh well, at least I believe it's well done.

Does CREDITS.TXT appear anywhere in the game? I want make sure it looks correctly. If not, I guess it's fine.

[dk]

I think you would not be able to upload it to Transifex, hence the different number of lines.

A solution would be to get rid of the many line breaks in this file, e.g.

The Dungeon Crawl Stone Soup team (Peter Berger, Adam Borowski, Samuel Bronson, Jude Brown, Robert Burnham, Chris Campbell, Matthew Cline, Paul Du Bois, Florian Diebold, Michael agno, Eino Keskitalo, Vsevolod...

instead of

The Dungeon Crawl Stone Soup team (Peter Berger, Adam Borowski, Samuel
Bronson, Jude Brown, Robert Burnham, Chris Campbell, Matthew Cline,
Paul Du Bois, Florian Diebold, Michael Gagno, Eino Keskitalo, Vsevolod...

since this is why files editors/viewers have automaic linebreaks.

[TehDruid]

I've run into some sort of problem. I'm doing this on Windows, remember. I started editing aptitudes.txt and the characters don't appear in game. Instead they are all question marks. Is this font-related or...? My system locale is set to Greek.

EDIT:

Nevermind, locale shouldn't play any part at this since I did this in UTF-8 and Crawl supports Unicode, right? I suppose it's because of the font that doesn't support (that's a guess) greek characters?

@dk: The main issue comes from the fact that some words don't translate into a single word in other languages or they do but that word is bigger than the origianl one or you have to explain it with a phrase instead and you end up taking a lot of space if you want people to understand what they should understand. I've also run into this problem while editing aptitudes.txt and I suppose this is gonna be a problem because of the way you navigate through Instructions in-game.

EDIT:

For now, I'll wait till Transifex is set up and I understand how it works, I guess. I'll check on this topic later today.

[JeffQyzt]

...Crawl supports Unicode, right?

Check your "char_set" value in your init.txt. It's set to ASCII by default.

[TehDruid]

That didn't do anything, hm...

[CommanderC]

If you are using Notepad++, open the "Encoding" menu, and make sure that "Encode in UTF-8" is selected.

[edlothiol]

Only the console and web versions support Unicode at the moment; local tiles doesn't. (char_set is, as far as I know, irrelevant to this; I believe it just configures the glyphs in the dungeon view.)

[XuaXua]

Sometimes, once you save a file in a certain fashion, you can't un encode it back to UTF-8, no matter what you do. It's weird, and it happens with some editors.

[galehar]

I think you would not be able to upload it to Transifex, hence the different number of lines.

A solution would be to get rid of the many line breaks in this file, e.g.

The Dungeon Crawl Stone Soup team (Peter Berger, Adam Borowski, Samuel Bronson, Jude Brown, Robert Burnham, Chris Campbell, Matthew Cline, Paul Du Bois, Florian Diebold, Michael agno, Eino Keskitalo, Vsevolod...

instead of

The Dungeon Crawl Stone Soup team (Peter Berger, Adam Borowski, Samuel
Bronson, Jude Brown, Robert Burnham, Chris Campbell, Matthew Cline,
Paul Du Bois, Florian Diebold, Michael Gagno, Eino Keskitalo, Vsevolod...

since this is why files editors/viewers have automaic linebreaks.

Most of our files have lines break manually at 80 chars. Personally, I hate it and prefer to rely on editor to do the wrapping automatically, but I couldn't convince the other guys. They seem to think that automatic wrapping is a rare and weird feature for some reason.
We'll have to figure out a way for transifex to handle translated files of different length, because english is a concise language so most translations will be longer. Shouldn't we translate strings instead of files?

[XuaXua]

If you want to do alternate languages, you have to do away with the manual wrapping and let the application handle it.

[Grimm]

XuaXua it sounds like you have a lot of experience with this sort of thing. I'd really appreciate any input or contributions you would like to make, especially with the use of Transifex or some more appropriate framework.

[jpeg]

I've had a look at the credits file uploaded to Transifex, but I can't make heads or tails of what I'm supposed to. If I "view the strings online" or "download for use", I see a lot of XML tags, but nothing that remotely resembles anything workable.

The Transifex help mentions a "Translate Now!" button (no screenshots, unfortunately) that so far I've been unable to find. The help page also links to a youtube video which appears to be out-of-date. So am I being dense here, do I need more rights, is the file mis-formated, ...?

Oh, I found the "Translate Now!" button under "CREDITS.txt 0.10 » German" and this brings up the columnized page that shows up on the Help page, but it's still all XML. On page 3, finally strings such as

<td class="code"><pre class="prettyprint lang-txt">* Brent Ross, Dungeon Crawl 4.0/4.1 maintainer. Stone Soup owes a lot to his</pre></td>

start appearing. Is this supposed to look like this? If not, what am I doing wrong?

Also, bigyellowcactus told me he'd done a translation, but I can't find it anywhere. However, when I eventually did find the "web-based translation editor" (though the translation column was initially empty) and clicked on one entry for sport, the translation memory popped up showing that indeed he had translated stuff.

I'm confused.

[MyOtheHedgeFox]

Most of our files have lines break manually at 80 chars. Personally, I hate it and prefer to rely on editor to do the wrapping automatically, but I couldn't convince the other guys. They seem to think that automatic wrapping is a rare and weird feature for some reason.

Use a very rare symbol for NOLINEBREAKHERE (as in "not&nbsp;applicable"). I think it should resolve the main problem of automatic wrapping I know of. =)
This is, actually, not that new in games. Daniel Remar used something close to that in Iji: in custom font, the $ character was displayed as a whitespace, thus becoming a non-breaking spacebar character. He also used it for aligning the text manually.

And -- yes, looks like we WILL need automatic wrapping, at least for the purpose of "graceful failure". We still can re-compose the text with manual caret returns, but for the ease of testing...



By the way, guys: what if we just upload the whole stable branch to an ftp server and point Transifex to it?

Also, could anyone tell me how to view the files in Gitorious without the fancy GUI (text-only)? Is there any quick link usable in a browser that displays a text file, not a fully-formed webpage with this text inside?
I believe that is what Transifex wants us to feed him...

Sometimes, once you save a file in a certain fashion, you can't un encode it back to UTF-8, no matter what you do. It's weird, and it happens with some editors.

What if you enter a certain non-Latin-charset character there, like ё, save in UTF-8, delete the symbol and save it again?

[TehDruid]

Oh, I've been running tiles, that's why it doesn't display right. Thanks for making this clear. I am using Notepad++ and yes, I made sure to have UTF-8 encoding enabled.

[Grimm]

jpeg: I changed the gitorious link for credits.txt as suggested by dk above and I think that deleted his German translation and gave us the new html-y display. That's why the messup.

I agree that their help files are not very clear or helpful.

[galehar]

For help, try to online chat. They seem responsive at least.

[Grimm]

This video has a slightly clearer explanation of how to use Transifex.

Pros and Cons:

Transifex pros:

dedicated translation environment - smooth interface
keeps project out of devs' hair until final stages

Transifex cons:

not immediately clear what the best way to use it is - help docs not great
not yet clear how to plug translations seamlessly back into Crawl
is yet ANOTHER place for Crawl activity

Dev wiki pros:

already integrated into Crawl coding
solves the "too many places" problem
devs can keep close eye on translation activity

Dev wiki cons:

would take some work to make it suitable for translation work
interface is not smooth
more noise for devs uninterested in translation

edit: I confused Mantis and Dev wiki

[jejorda2]

So is the odd Irishy dialect translation on CDO an April Fools joke or an effort to test the translation system?

Is there an option or in game command to switch between translations?

ACK! The faux Irish was silly, but this poor Yiddish thing is killing me!

[galehar]

So is the odd Irishy dialect translation on CDO an April Fools joke or an effort to test the translation system?

Is there an option or in game command to switch between translations?

ACK! The faux Irish was silly, but this poor Yiddish thing is killing me!

Both. It's an early April fool joke, and a way to test the translation system. And it's not Irish, it's Durkon's speech (from OOTS. And it's like Scottish accent). There's also Jäger's speech from girl genius.