Add CSV file to crawl

[Marbit]

Problem: Crawl morgue files are human readable but a mess to parse, present the data in an uncondensed format, different morgue files are difficult to compare and the history of the character is somewhat incomplete.

Solution: Write the information in comma separated file with data from character, every time the character reaches some amount of experience points both the file and the morgue of the character are appended with the line that has the information in a comma separated format.

Benefits: With the comma separated file players could quickly and easily see things like which species is most played, in which levels do they die the most, at which levels do their transmuters die or compare the development (not only the dead result) of several characters of a kind.

I imagine advisors checking the files and discovering things that are usually difficult to see with one morgue file from a character that reached high level and that is still alive:

-Most low level characters (between level 2 and 6) have a traps&doors skill above 3.
-All of the characters between experience level 8 and 14 die at orc dungeon 4.
-The player isn't quivering anything with characters of level bellow experience level 6.
-90% of the characters from this player die before they reach level 4.
-This player plays mostly hill orcs (85% of characters), the ones that are priests survive on average more that the healers.

Example of a CSV file:

  Code:

name,species,class,god,exp,lvl,arm,ddg,stb,location,spell1
Purg,Tr,Be,Trog,5000,8,8.2,10,6,D10,,
Purg,Tr,Be,The_Shining_One,15000,10,14.6,15,8,L1,blink,
John,HO,Tm,,3023,2,2,3,0,D2,beastly_apendage


For anyone not familiar with comma separated value files just open a text editor (wordpad,notepad,gedit...) paste the four lines of the example onto a blank file, save it in .csv format and open it with an excel like program it (if it ask you to specify a separator, specify commas).

These kind of files are read by most statistical packages and excel like programs and will provide a lot of information in a very little space. Also I think it shouldn't be very difficult to implement (for people that know a bit of crawl code I mean).

Pseudocode example:

  Code:

if (condition){
echo -e $name,$species,$class,$god,etc... >> morgue_file_of the character
echo -e $name,$species,$class,$god,etc... >> file_for_all_characters
}

The pseudocode above is more kind of a shell script (prinf or cout for c++?) and surely it is an oversimplificacion but I hope that it makes the point that the only thing that I'm asking for is for crawl to append the morgue file with the values of a character separated by a comma and to append another file with the same data. Also it could be nice if there were no spaces in the names (some programs get crazy with them) so that "The Shining One" would appear as "The_Shining_One"

Edit: Edited for cosmetic changes.

[Marbit]

Another example of what you can get with the data, a distribution of the level of my characters when they died or escaped the dungeon.


Do you want to see yours? Or the porcentage of characters of each race that you played?

[Moose]

Not a bad idea, but if this data were to be made available in a format for parsing by code, I'd recommend something other (and less problematic) than CSV, like YAML, or preferably Google' protocol buffer format.

[Marbit]

Not a bad idea, but if this data were to be made available in a format for parsing by code, I'd recommend something other (and less problematic) than CSV, like YAML, or preferably Google' protocol buffer format.

I've never used YAML or google protocol buffer format, I'll have to check them. Can excel or statistical software like R open such files? Can you post the examples that I wrote in that format so that I can compare them?

[Moose]

afaik, Google provides protocol buffer compilers for Go, C++, java, and Python, and I know that libraries exist in Perl. Any language cam also implement their own pb compilers if need be.

YAML libraries exist for pretty much every mainstream language out there. protocol buffers are easy to deal with in code (just getters and setters), cross compile, and serialize in an extremely compact format.

I think it could be cool if morgue files were generated in a serialized PB format, an then have a script that reads those, parses them out, and displays the human-readable morgue file based on the data in the PB.

Here's a nice comparison of data-serialization formats: http://en.wikipedia.org/wiki/Comparison ... on_formats

[mumra]

Wow, protocol buffer looks cool. Hadn't seen that before. I wonder if we could be using this for anything else, e.g. webtiles data packets ...

[Marbit]

Checking YAML it seems useful to manage and display things like egos in artifacts, the player's inventory, the set of mutations and status that the player has and properties of different objects. Being able to access the properties of hundreds of objects might be nice for a database and to answer questions like:

Which characters had mutation X and Y?
Which characters had a weapon with +stealth and +blink?

The YAML would enable us to refine the search to the properties of objects but If we want to check general stuff like how characters train their skills, which proportion of characters are of a certain race or background, a table like format will do. For that we better stick with the good old comma separated values format (CSV). It is strong and widely used by statisticians and people in science, requires no packages or libraries to be processed and can be read and managed by people with no programming skills in any excel like program. What I'm trying to say is: don't bring the lasers where a screwdriver will do.

Edit: Format.

[neil]

If you compile with -DDGL_MILESTONES you already get some of this information in an easy-to-parse (colon-separated) form. This is where Sequell and the other bots in ##crawl get most of their data. See for example (Warning: these files are 77 MB and 231 MB, respectively!):

Games: http://dobrazupa.org/meta/git/logfile
Important in-game events: http://dobrazupa.org/meta/git/milestones

[brendan]

Wow, protocol buffer looks cool. Hadn't seen that before. I wonder if we could be using this for anything else, e.g. webtiles data packets ...

protobuf is a reasonable format to target. It has wide use, tooling support and it's BSD licensed.

It probably isn't worth the effort, but maybe we should consider using it as a save file target. I think it would give us stricter guarantees around breaking save compatibility and facilitate downstream users from mucking with save files if they wanted (Haven't you ever wanted to import a crawl dungeon into your dwarf fortress game? ).

For a simultaneously human readable / machine parsable format, I prefer json. Luckily there's a protobuf -> json converter!

Edit: I'd like to state my opposition to CSV files. There are some ugly idiosyncrasies.

[njvack]

There are idiosyncrasies with CSV -- but as long as you choose one set, they're not so bad.

I think, though, that making sane, useful morgues into CSV files would suck; you'd want to nest lists which would be ugly. Protobuf/json would be nice.

[floatboth]

Would be nice if Crawl exported JSON or YAML.

But for now, I wrote a parser in Clojure… Check out clj-dcss

[Siegurt]

Here's the quick and dirty perl script I wrote to do it:

  Code:

open IN, "<scores";
$linenum = 0;
$colnum = 0;

while($line = <IN>)
{
   $linenum++;
   @things = split(':',$line);
   foreach $thing (@things)
   {
      ($type,$def) = $thing =~/(.*)=(.*)/;

      $results[$linenum]->{$type} = $def;
      if(!exists($coldef{$type}))
      {
         $colorder[$colnum] = $type;
         $coldef{$type} = $colnum++;
      }
   }
}

close IN;
open OUT, ">scores.csv";
foreach $col (@colorder)
{
   print OUT "$col,";
}
print OUT "\n";
foreach $res (@results)
{
   $classcnt{$res->{cls}}++;
   $racecnt{$res->{race}}++;
   $charcnt{$res->{char}}++;
   $godcnt{$res->{god}}++;
   $killercnt{$res->{killer}}++;
   $killersc{$res->{killer}}+=$res->{xl};
   $scorecnt{$res->{char}}+=$res->{sc};
   $scorecnt{$res->{cls}}+=$res->{sc};
   $scorecnt{$res->{race}}+=$res->{sc};
   $sccnt++;
   $sctot += $res->{sc};
   foreach $col (@colorder)
   {
      $res->{$col} =~ s/"/""/g;
      print OUT '"'.$res->{$col}.'"';
      print OUT ",";
   }
   print OUT "\n";
}
close OUT;
$avgsc = $sctot/$sccnt;
print "Average Score:".$avgsc."\n";
print "Avg by class:\n";
foreach $class (keys(%classcnt))
{
   print "\t$class: $classcnt{$class}/".$scorecnt{$class}/$classcnt{$class}."\n";
}
print "Avg by race:\n";
foreach $race (keys(%racecnt))
{
   print "\t$race: $racecnt{$race}/".$scorecnt{$race}/$racecnt{$race}."\n";
}
print "Avg by Race/Class combo:\n";
foreach $char (keys(%charcnt))
{
   print "\t$char: $charcnt{$char}/".$scorecnt{$char}/$charcnt{$char}."\n";
}
print "Gods:\n";
foreach $god (keys(%godcnt))
{
   print "\t$god: $godcnt{$god}\n";
}
print "Killers, average char level:\n";
foreach $killer (keys(%killercnt))
{
   print "\t$killer: $killercnt{$killer}/$killersc{$killer}\n";
}
print "Press <ENTER> to close\n";
<STDIN>;