Page 2 of 6
Re: Translations
Posted:
Tuesday, 27th March 2012, 21:21
by XuaXua
MyOtheHedgeFox wrote:If you export something via Transifex/Launchpad/etc, it almost always gives you a Unicode tex... hm, yes, Xua, you are right!
I have over a decade of I18N experience under my belt. The worst is working for companies who use the default database encoding.
Re: Translations
Posted:
Tuesday, 27th March 2012, 21:46
by mumra
XuaXua wrote:MyOtheHedgeFox wrote:If you export something via Transifex/Launchpad/etc, it almost always gives you a Unicode tex... hm, yes, Xua, you are right!
I have over a decade of I18N experience under my belt. The worst is working for companies who use the default database encoding.
That's a horrible problem I've run into before on mysql. I can't remember the exact context; basically there was a database with Swedish Latin (which is mysql's slightly bizarre default setting), but some tables had been created with UTF8, or something like that. Was a right mess sorting it all out!
Re: Translations
Posted:
Tuesday, 27th March 2012, 22:05
by XuaXua
mumra wrote:XuaXua wrote:MyOtheHedgeFox wrote:If you export something via Transifex/Launchpad/etc, it almost always gives you a Unicode tex... hm, yes, Xua, you are right!
I have over a decade of I18N experience under my belt. The worst is working for companies who use the default database encoding.
That's a horrible problem I've run into before on mysql. I can't remember the exact context; basically there was a database with Swedish Latin (which is mysql's slightly bizarre default setting), but some tables had been created with UTF8, or something like that. Was a right mess sorting it all out!
Since the database involved here is likely flat text files (akin to Java's "Resource Bundles"), the other primary issue I've encountered concerning multiple developers on the same I18N project (the Stone Soup, if you will, of editor applications being used to edit a text, seemingly with the proper encoding set) comes into play. I've fallen in the bad editor trap twice; once when I first discovered it, and another time when another developer took on I18N responsibility of his own accord, did a find and replace, and ruined a deployment. Rollback was easy, but consistency, rule enforcement, and training are keys to prevention.
Re: Translations
Posted:
Wednesday, 28th March 2012, 05:51
by MyOtheHedgeFox
I think focusing on these nifties is just losing time now...
Re: Translations
Posted:
Thursday, 29th March 2012, 01:17
by Grimm
I've set up a Transifex page for us here
https://www.transifex.net/projects/p/dcss/Interested translators please sign up and take a look. Let's evaluate it and decide whether it's what we need.
Re: Translations
Posted:
Thursday, 29th March 2012, 07:38
by dk
I'm not 100% sure how the workflow is/should be with Transifex, but would you mind invite me to the project (my username is bigyellowcactus)?
Re: Translations
Posted:
Thursday, 29th March 2012, 08:32
by TehDruid
You have my sword...
Ok. I'm in, now I have to see how this works. How should I start translating, first of all? Guess I have to get the source code first of all?
Re: Translations
Posted:
Thursday, 29th March 2012, 09:38
by MyOtheHedgeFox
If you translate a game, you should have:
1) Something you translate;
2) A tool to convert (1) into game-usable format;
3) A way to see how it looks in the game (source code, a special tool, daily "please make a new test version" mail to the game developers...).
If you are going to translate the game, GET THE SOURCE CODE.
You must be able to find which each string is used.
<A lot of blabber was cut out>
Re: Translations
Posted:
Thursday, 29th March 2012, 11:28
by TehDruid
I've compiled the game in Win7 some months ago cause I was trying to add some unrandarts of my own, no worries. I'll just have to freshen my memory.
So, basically, I find the appropriate piece of text in the source files, change it and then compile the game to see how it plays out. Perhaps this is a stupid question, but can't we translate without having to compile the game every time we want to check if our work appears correctly? That's going to take quite a while.
I'll give it a go after I'm back from class tomorrow morning, anyway.
Re: Translations
Posted:
Thursday, 29th March 2012, 12:04
by MyOtheHedgeFox
Erm... Saw my post above? That was a nice example of Yiuf-class nonsense caused by overstudy.
I beg your pardon for
all the nonsense I wrote in the previous message. I do not think it will work right now, so what you should do at the moment is translate all the strings using Grimm's Transifex Workshop.
Re: Translations
Posted:
Thursday, 29th March 2012, 12:23
by galehar
TehDruid wrote:I've compiled the game in Win7 some months ago cause I was trying to add some unrandarts of my own, no worries. I'll just have to freshen my memory.
So, basically, I find the appropriate piece of text in the source files, change it and then compile the game to see how it plays out. Perhaps this is a stupid question, but can't we translate without having to compile the game every time we want to check if our work appears correctly? That's going to take quite a while.
I'll give it a go after I'm back from class tomorrow morning, anyway.
If you want to test locally a change in descriptions or database, there is no need for compiling. Just edit the file in dat/descript or dat/database and restart the game. But you should submit your translations or improved english descriptions on transiflex once the project is properly set up.
Re: Translations
Posted:
Thursday, 29th March 2012, 12:27
by TehDruid
@MyO: Very well. I'll have to converse with Grimm then. Btw I didn't find your previous post annoying or anything.
@Galehar: Thank you!
Re: Translations
Posted:
Thursday, 29th March 2012, 12:28
by jpeg
TehDruid wrote:How should I start translating, first of all? Guess I have to get the source code first of all?
Seeing how it's currently just descriptions that are being translated, you shouldn't require the source code at all. If you need to check how your translation looks in-game (which might be important to see how much fits the screen in a 24x80 window or whether special characters are getting displayed correctly) you can directly copy/paste your translation into the corresponding description file (in the dat/descript/ directory).
Later on, when dynamic messages (even stuff like speech) get translated, checking the results in-game will become more important, but even then compiling
shouldn't be necessary, seeing how all messages are being outsourced to the dat/ directory.
EDIT: That's what I get for taking so long to compose my posts, and getting interrupted by actual work...
Re: Translations
Posted:
Thursday, 29th March 2012, 14:12
by CommanderC
Right now the main obstacle I see is that we need to import all those files in dat/descript into transifex. I don't know about the status of galehar's script, but I have made this (ugly) python script that maybe can be handy. It transforms the .txt files into .yaml files. It should be run in the root directory of crawl (the one that contains source, docs, etc)
- Code:
#!/usr/bin/env python
import os
import os.path
import yaml
def parse(input_file):
with open(input_file) as f:
state = 0
key = ""
value = []
for l in f:
if l[0] == '#' or len(l) == 1:
pass
elif l[0:4] == '%%%%':
if state == 1:
yield key, ''.join(value)
state = 2
elif state == 2:
key = l
value = []
state = 1
elif state == 1:
value.append(l)
def process(input_file, output_file, locale):
with open(output_file, 'w') as f:
data = {}
for k,v in parse(input_file):
data[k] = v
result = {locale:data}
yaml.dump(result, f)
def main():
output_dir = 'output'
input_dir = os.path.join('source', 'dat', 'descript')
files = ['ability', 'branches', 'commands', 'gods', 'monsters', 'skills', 'spells', 'unrand', 'backgrounds', 'cards', 'features', 'items', 'species', 'unident']
all_locales = ['de', 'es', 'ru']
for locale in all_locales:
for f in files:
output_dir2 = os.path.join(output_dir, locale)
if not os.path.exists(output_dir2):
os.makedirs(output_dir2)
output_file = os.path.join(output_dir, locale, f + '.yaml')
input_file = os.path.join(input_dir, f + '.txt')
process(input_file, output_file, locale)
if __name__ == '__main__':
main()
Re: Translations
Posted:
Thursday, 29th March 2012, 14:26
by galehar
Here it is. Not much prettier than yours
Should be easy to adapt it to make it spit yaml output instead of dokuwiki.
Right now all our scripts in util are written in perl, and kilobyte (who is doing translations) use perl for scripting. So if the export script were to be commited in trunk, it would be better if it were written in perl.
Python is fine too, if you guys polish a python script and maintain it, we won't turn it down.
- Code:
#!/usr/bin/perl
use strict;
use warnings;
use Text::Wrap;
$Text::Wrap::columns = 80;
$Text::Wrap::separator = " \\\\ ";
my $base_dir = `git rev-parse --show-toplevel`;
chomp $base_dir;
#my $quotes = "$base_dir/crawl-ref/source/dat/database/quotes.txt";
#die "Cannot open $quotes for reading\n" unless (-r $quotes);
die "Usage: $0 description_files\n" unless (@ARGV);
my %DESCRIPTIONS;
#my %QUOTES;
main();
sub main {
# load_file($quotes, \%QUOTES);
foreach (@ARGV) {
load_file($_, \%DESCRIPTIONS);
}
gen_array();
}
sub load_file {
my ($input, $hash) = @_;
open IN, $input or die "Cannot open $input";
my ($key, $value);
while(<IN>) {
next if (/^#/ or /^\s*$/);
chomp;
if (/^%%%%$/) {
if ($key) {
$value =~ s/([{}])/\\$1/g;
$hash->{$key} = wrap("", "", $value);
}
$key = "";
$value = "";
next;
}
if ($key) {
$value .= $_;
$value .= " \\\\ " if /^\s+/;
} else {
$key = $_;
}
}
close IN;
}
sub gen_array {
print "^Original^^Translation^^ ^ ^\n";
print "^Name^Description^Name^Description^Review^Notes^\n";
foreach (sort keys %DESCRIPTIONS) {
printf "|%s|%s| | | |\n", $_, $DESCRIPTIONS{$_};
}
}
Re: Translations
Posted:
Thursday, 29th March 2012, 15:40
by CommanderC
Thanks, galehar. I have modified your script. Here it is:
- Code:
#!/usr/bin/perl
use strict;
use warnings;
use YAML ();
#local $YAML::SortKeys = 0;
#use Text::Wrap;
#$Text::Wrap::columns = 80;
#$Text::Wrap::separator = " \\\\ ";
my $base_dir = `git rev-parse --show-toplevel`;
chomp $base_dir;
#my $quotes = "$base_dir/crawl-ref/source/dat/database/quotes.txt";
#die "Cannot open $quotes for reading\n" unless (-r $quotes);
die "Usage: $0 description_files\n" unless (@ARGV);
my %DESCRIPTIONS;
#my %QUOTES;
main();
sub main {
# load_file($quotes, \%QUOTES);
foreach (@ARGV) {
load_file($_, \%DESCRIPTIONS);
}
gen_array();
}
sub load_file {
my ($input, $hash) = @_;
open IN, $input or die "Cannot open $input";
my ($key, $value);
while(<IN>) {
next if (/^#/ or /^\s*$/);
chomp;
if (/^%%%%$/) {
if ($key) {
$value =~ s/([{}])/\\$1/g;
# $hash->{$key} = wrap("", "", $value);
$hash->{$key} = $value;
}
$key = "";
$value = "";
next;
}
if ($key) {
$value .= $_;
$value .= " \\\\ " if /^\s+/;
} else {
$key = $_;
}
}
close IN;
}
sub gen_array {
my $data = [];
my $i = 0;
foreach (sort keys %DESCRIPTIONS) {
$data->[$i++] = {
"Original Name" => $_,
"Original Description" => $DESCRIPTIONS{$_},
"Translated Name" => "",
"Translated Description" => "",
"Review" => "",
"Notes" => ""
};
}
print YAML::Dump({en => $data});
}
Re: Translations
Posted:
Thursday, 29th March 2012, 16:55
by Grimm
Let's slow down a moment and make sure Transifex is going to work. I have no experience with this sort of program. I am currently unable to figure out how to import text into Transifex - the documentation says that we can input a gitorious link to transfer text but I can't find the page that allows that. I also don't see how I can give people "maintainer" status.
Also, if you can use your Crawl name there that would help us keep clear who is who.
Re: Translations
Posted:
Thursday, 29th March 2012, 19:15
by MyOtheHedgeFox
Once you created a language group, the one who requested its creation automatically becomes its maintainer.
Re: Translations
Posted:
Thursday, 29th March 2012, 22:43
by Grimm
I've entered CREDITS.txt from 0.10.1 as a resource on Transifex, let's try translating that to see how it goes.
Now I see where the URL can go. Is
http://gitorious.org/crawl/crawl/blobs/ ... REDITS.txtthe location of the 0.10.1 version? That's what I've linked to at the moment.
Re: Translations
Posted:
Thursday, 29th March 2012, 23:09
by Kate
Re: Translations
Posted:
Friday, 30th March 2012, 08:01
by dk
German version of the credits added! Yeah!
OK, took only some minutes...
FYI: If working with plaintext files, the translated file has to have the same number of lines as the original. That sucks
Re: Translations
Posted:
Friday, 30th March 2012, 08:38
by dk
OK, it seems the Changelog.txt-file changed. Now there's a html-page from gitorious instead of the actual text-file and my translations is lost...
Re: Translations
Posted:
Friday, 30th March 2012, 08:41
by dk
Re: Translations
Posted:
Friday, 30th March 2012, 08:43
by TehDruid
I opened the current file that is on Transifex on Notepad++ and it contains html code instead of just text. So I'm kinda clueless on what to translate and what to not translate of it. Any advice would be appreciated.
Re: Translations
Posted:
Friday, 30th March 2012, 08:59
by dk
TehDruid wrote:I opened the current file that is on Transifex on Notepad++ and it contains html code instead of just text. So I'm kinda clueless on what to translate and what to not translate of it. Any advice would be appreciated.
It seems to be a problem with the url of the file. Transifex seems to check out the file automatically, but the url links to the gitorious page instead of the actual file
Re: Translations
Posted:
Friday, 30th March 2012, 09:03
by TehDruid
Should I wait for that to be corrected then, or download 0.10 (or is it 0.10.1 we're working on? Is there a difference in text between those?) and translate it locally then upload it to Transifex?
Re: Translations
Posted:
Friday, 30th March 2012, 09:12
by dk
You can download it here:
http://gitorious.org/crawl/crawl/blobs/ ... REDITS.txtBut before uploading, we have to wait 'till this issue is fixed.
Re: Translations
Posted:
Friday, 30th March 2012, 09:21
by Grimm
I've changed it to
http://gitorious.org/crawl/crawl/blobs/ ... REDITS.txt but now it shows all the html. Perhaps it would be simpler to upload the .txt files by hand.
Re: Translations
Posted:
Friday, 30th March 2012, 09:32
by dk
Maybe it is because Transifex did not pull the file directly after you changed the url. Is there a way to force Transifex to pull the file from git?
This morning I downloaded the credits.txt with no problems, uploaded my translation, and some minutes later I wondered why my translation was lost. I noticed the the time that is displayed right next to the resource file changed to Mar 30, 08:30a.m. This seems to be the point in time where Transifex synchronized the files and downloaded the html-page, which then overwrote the real credits.txt file uploaded by hand earlier.
Re: Translations
Posted:
Friday, 30th March 2012, 09:55
by MyOtheHedgeFox
Is there a way to access the git repo as if it was an ftp server?
Re: Translations
Posted:
Friday, 30th March 2012, 09:59
by dk
you cann access the git repo via http/https:
http://gitorious.org/crawl
Re: Translations
Posted:
Friday, 30th March 2012, 10:11
by TehDruid
I'm done with CREDITS.TXT although I had to make it some lines longer because I couldn't keep within the limits of each line. Oh well, at least I believe it's well done.
Does CREDITS.TXT appear anywhere in the game? I want make sure it looks correctly. If not, I guess it's fine.
Re: Translations
Posted:
Friday, 30th March 2012, 10:17
by dk
I think you would not be able to upload it to Transifex, hence the different number of lines.
A solution would be to get rid of the many line breaks in this file, e.g.
The Dungeon Crawl Stone Soup team (Peter Berger, Adam Borowski, Samuel Bronson, Jude Brown, Robert Burnham, Chris Campbell, Matthew Cline, Paul Du Bois, Florian Diebold, Michael agno, Eino Keskitalo, Vsevolod...
instead of
The Dungeon Crawl Stone Soup team (Peter Berger, Adam Borowski, Samuel
Bronson, Jude Brown, Robert Burnham, Chris Campbell, Matthew Cline,
Paul Du Bois, Florian Diebold, Michael Gagno, Eino Keskitalo, Vsevolod...
since this is why files editors/viewers have automaic linebreaks.
Re: Translations
Posted:
Friday, 30th March 2012, 11:11
by TehDruid
I've run into some sort of problem. I'm doing this on Windows, remember. I started editing aptitudes.txt and the characters don't appear in game. Instead they are all question marks. Is this font-related or...? My system locale is set to Greek.
EDIT:
Nevermind, locale shouldn't play any part at this since I did this in UTF-8 and Crawl supports Unicode, right? I suppose it's because of the font that doesn't support (that's a guess) greek characters?
@dk: The main issue comes from the fact that some words don't translate into a single word in other languages or they do but that word is bigger than the origianl one or you have to explain it with a phrase instead and you end up taking a lot of space if you want people to understand what they should understand. I've also run into this problem while editing aptitudes.txt and I suppose this is gonna be a problem because of the way you navigate through Instructions in-game.
EDIT:
For now, I'll wait till Transifex is set up and I understand how it works, I guess. I'll check on this topic later today.
Re: Translations
Posted:
Friday, 30th March 2012, 11:50
by JeffQyzt
...Crawl supports Unicode, right?
Check your "char_set" value in your init.txt. It's set to ASCII by default.
Re: Translations
Posted:
Friday, 30th March 2012, 12:54
by TehDruid
That didn't do anything, hm...
Re: Translations
Posted:
Friday, 30th March 2012, 14:05
by CommanderC
If you are using Notepad++, open the "Encoding" menu, and make sure that "Encode in UTF-8" is selected.
Re: Translations
Posted:
Friday, 30th March 2012, 14:18
by edlothiol
Only the console and web versions support Unicode at the moment; local tiles doesn't. (char_set is, as far as I know, irrelevant to this; I believe it just configures the glyphs in the dungeon view.)
Re: Translations
Posted:
Friday, 30th March 2012, 14:36
by XuaXua
Sometimes, once you save a file in a certain fashion, you can't un encode it back to UTF-8, no matter what you do. It's weird, and it happens with some editors.
Re: Translations
Posted:
Friday, 30th March 2012, 14:58
by galehar
dk wrote:I think you would not be able to upload it to Transifex, hence the different number of lines.
A solution would be to get rid of the many line breaks in this file, e.g.
The Dungeon Crawl Stone Soup team (Peter Berger, Adam Borowski, Samuel Bronson, Jude Brown, Robert Burnham, Chris Campbell, Matthew Cline, Paul Du Bois, Florian Diebold, Michael agno, Eino Keskitalo, Vsevolod...
instead of
The Dungeon Crawl Stone Soup team (Peter Berger, Adam Borowski, Samuel
Bronson, Jude Brown, Robert Burnham, Chris Campbell, Matthew Cline,
Paul Du Bois, Florian Diebold, Michael Gagno, Eino Keskitalo, Vsevolod...
since this is why files editors/viewers have automaic linebreaks.
Most of our files have lines break manually at 80 chars. Personally, I hate it and prefer to rely on editor to do the wrapping automatically, but I couldn't convince the other guys. They seem to think that automatic wrapping is a rare and weird feature for some reason.
We'll have to figure out a way for transifex to handle translated files of different length, because english is a concise language so most translations will be longer. Shouldn't we translate strings instead of files?
Re: Translations
Posted:
Friday, 30th March 2012, 17:05
by XuaXua
If you want to do alternate languages, you have to do away with the manual wrapping and let the application handle it.
Re: Translations
Posted:
Friday, 30th March 2012, 17:44
by Grimm
XuaXua it sounds like you have a lot of experience with this sort of thing. I'd really appreciate any input or contributions you would like to make, especially with the use of Transifex or some more appropriate framework.
Re: Translations
Posted:
Friday, 30th March 2012, 18:57
by jpeg
I've had a look at the credits file uploaded to Transifex, but I can't make heads or tails of what I'm supposed to. If I "view the strings online" or "download for use", I see a lot of XML tags, but nothing that remotely resembles anything workable.
The Transifex help mentions a "Translate Now!" button (no screenshots, unfortunately) that so far I've been unable to find. The
help page also links to a youtube video which appears to be out-of-date. So am I being dense here, do I need more rights, is the file mis-formated, ...?
Oh, I found the "Translate Now!" button under "CREDITS.txt 0.10 » German" and this brings up the columnized page that shows up on the Help page, but it's still all XML. On page 3, finally strings such as
<td class="code"><pre class="prettyprint lang-txt">* Brent Ross, Dungeon Crawl 4.0/4.1 maintainer. Stone Soup owes a lot to his</pre></td>
start appearing. Is this supposed to look like this? If not, what am I doing wrong?
Also, bigyellowcactus told me he'd done a translation, but I can't find it anywhere. However, when I eventually did find the "web-based translation editor" (though the translation column was initially empty) and clicked on one entry for sport, the translation memory popped up showing that indeed he had translated stuff.
I'm confused.
Re: Translations
Posted:
Friday, 30th March 2012, 19:08
by MyOtheHedgeFox
galehar wrote:Most of our files have lines break manually at 80 chars. Personally, I hate it and prefer to rely on editor to do the wrapping automatically, but I couldn't convince the other guys. They seem to think that automatic wrapping is a rare and weird feature for some reason.
Use a very rare symbol for NOLINEBREAKHERE (as in "not
applicable"). I think it should resolve the main problem of automatic wrapping I know of. =)
This is, actually, not that new in games. Daniel Remar used something close to that in Iji: in custom font, the $ character was displayed as a whitespace, thus becoming a non-breaking spacebar character. He also used it for aligning the text manually.And -- yes, looks like we WILL need automatic wrapping, at least for the purpose of "graceful failure". We still can re-compose the text with manual caret returns, but for the ease of testing...
By the way, guys: what if we just upload the whole stable branch to an ftp server and point Transifex to it?
Also, could anyone tell me how to view the files in Gitorious without the fancy GUI (text-only)? Is there any quick link usable in a browser that displays a text file, not a fully-formed webpage with this text inside?
I believe that is what Transifex wants us to feed him...
XuaXua wrote:Sometimes, once you save a file in a certain fashion, you can't un encode it back to UTF-8, no matter what you do. It's weird, and it happens with some editors.
What if you enter a certain non-Latin-charset character there, like
ё, save in UTF-8, delete the symbol and save it again?
Re: Translations
Posted:
Friday, 30th March 2012, 19:26
by TehDruid
Oh, I've been running tiles, that's why it doesn't display right. Thanks for making this clear. I am using Notepad++ and yes, I made sure to have UTF-8 encoding enabled.
Re: Translations
Posted:
Friday, 30th March 2012, 19:31
by Grimm
jpeg: I changed the gitorious link for credits.txt as suggested by dk above and I think that deleted his German translation and gave us the new html-y display. That's why the messup.
I agree that their help files are not very clear or helpful.
Re: Translations
Posted:
Friday, 30th March 2012, 20:22
by galehar
For help, try to online chat. They seem responsive at least.
Re: Translations
Posted:
Friday, 30th March 2012, 20:24
by Grimm
This video has a slightly clearer explanation of how to use Transifex.
Pros and Cons:Transifex pros:
dedicated translation environment - smooth interface
keeps project out of devs' hair until final stages
Transifex cons:
not immediately clear what the best way to use it is - help docs not great
not yet clear how to plug translations seamlessly back into Crawl
is yet ANOTHER place for Crawl activity
Dev wiki pros:
already integrated into Crawl coding
solves the "too many places" problem
devs can keep close eye on translation activity
Dev wiki cons:
would take some work to make it suitable for translation work
interface is not smooth
more noise for devs uninterested in translation
edit: I confused Mantis and Dev wiki
Re: Translations
Posted:
Saturday, 31st March 2012, 11:06
by jejorda2
So is the odd Irishy dialect translation on CDO an April Fools joke or an effort to test the translation system?
Is there an option or in game command to switch between translations?
ACK! The faux Irish was silly, but this poor Yiddish thing is killing me!
Re: Translations
Posted:
Saturday, 31st March 2012, 12:39
by galehar
jejorda2 wrote:So is the odd Irishy dialect translation on CDO an April Fools joke or an effort to test the translation system?
Is there an option or in game command to switch between translations?
ACK! The faux Irish was silly, but this poor Yiddish thing is killing me!
Both. It's an early April fool joke, and a way to test the translation system. And it's not Irish, it's Durkon's speech (from OOTS. And it's like Scottish accent). There's also Jäger's speech from girl genius.