Page 1 of 1

DCSS stat utility

PostPosted: Tuesday, 3rd October 2017, 13:54
by lepoulpe303
Hi all,

I've created a tool (in alpha stage ...) to gather stats from morgue files.
I hope this project will both improve my python and DCSS skills ..
An example of output of this tool is here : https://gist.github.com/oschwab/850fad890bcd1ce6ab24582a14a81385
Scores can be exported to csv and then in spreadsheet :https://i.imgur.com/3gQbqkB.png

I'm looking for

    * people with thick morgue folder, who are interested in sending me their morgue and see the result (I've never escaped with the orb, so I expect some troubles when parsing morgues of tough players)

    * people with python installed (on any OS)
    or
    * people with good DCSS knowledge


to help me improve statistics, and the program itself.

The project is on Git here https://github.com/oschwab/dcss_stats

Thanks for your attention.

Re: DCSS stat utility

PostPosted: Tuesday, 3rd October 2017, 16:52
by braveplatypus
I downloaded my online webtiles morgue files and uploaded them to google drive, it's a pretty thick folder but the files seem to be saved slightly differently then the way the offline version saves them, i hope it's not a problem.
Link

Re: DCSS stat utility

PostPosted: Tuesday, 3rd October 2017, 18:13
by lepoulpe303
6 first files, already 3 "issues" ... don"t expect the final result too quickly ;)

Re: DCSS stat utility

PostPosted: Tuesday, 3rd October 2017, 19:39
by braveplatypus
I just realized, that first folder is only the games i played on one of the servers, since games from different online servers are saved separately, here is my other online morgue folder.

Ready for more headache?

This server saves the files in the offline style :lol:

On the other hand maybe that means less errors.

If you end up not being able to match the morgues or cant get data from the other save style don't worry about it.

Also, some of these games are from much much older versions, and some are from fresh versions, that might be the cause of some errors. Feel free to clip out super old games if that helps.

Re: DCSS stat utility

PostPosted: Tuesday, 3rd October 2017, 23:04
by chequers
You can download all morgues from all online servers pretty easily. Here's the link for CPO for example: https://crawl.project357.org/morgue/

Re: DCSS stat utility

PostPosted: Wednesday, 4th October 2017, 01:28
by advil
It may or may not help you, because the code is a bit of a mess and doesn't get updated that much, but dcss actually has a built-in morgue parser that you could use as a reference, in wiz-dump.cc. This is used to try to construct a character from a morgue file for testing or fsim purposes in wizmode. (Parsing these in python would be a lot better than in c++!)

I think the online vs. offline morgue discussion in the above comments is a slight red herring: there's a whole bunch of options that can affect your morgue file and IIRC they have different default settings online vs. offline, or maybe per server. But the settings all work for both cases, so the source of the morgue doesn't really determine the details of the morgue, and the format is (unfortunately for someone trying to parse it) quite flexible. See section 4 of the options guide, especially the dump_order setting.

chequers wrote:You can download all morgues from all online servers pretty easily. Here's the link for CPO for exampl


uh if you do this, probably check with the server admin, it is sort of easy to accidentally DOS the server (as I understand it basically because the directory listings are really cpu intensive to generate), which can impact crawl games on the server. Usually throttling your downloader some is enough. (chequers is CPO admin so I guess you're ok for that one?)

Re: DCSS stat utility

PostPosted: Wednesday, 4th October 2017, 08:16
by lepoulpe303
braveplatypus wrote:I downloaded my online webtiles morgue files and uploaded them to google drive, it's a pretty thick folder but the files seem to be saved slightly differently then the way the offline version saves them, i hope it's not a problem.
Link

First draft is ready.

Apparently no private messages possible on this board.
Do you want me to put it in a public repository ?

Re: DCSS stat utility

PostPosted: Wednesday, 4th October 2017, 08:22
by lepoulpe303
advil wrote:It may or may not help you, because the code is a bit of a mess and doesn't get updated that much, but dcss actually has a built-in morgue parser that you could use as a reference, in wiz-dump.cc. This is used to try to construct a character from a morgue file for testing or fsim purposes in wizmode. (Parsing these in python would be a lot better than in c++!)


I simply can't C++. I avoid it even at work.

I think the online vs. offline morgue discussion in the above comments is a slight red herring: there's a whole bunch of options that can affect your morgue file and IIRC they have different default settings online vs. offline, or maybe per server. But the settings all work for both cases, so the source of the morgue doesn't really determine the details of the morgue, and the format is (unfortunately for someone trying to parse it) quite flexible. See section 4 of the options guide, especially the dump_order setting.

chequers wrote:You can download all morgues from all online servers pretty easily. Here's the link for CPO for exampl

uh if you do this, probably check with the server admin, it is sort of easy to accidentally DOS the server (as I understand it basically because the directory listings are really cpu intensive to generate), which can impact crawl games on the server. Usually throttling your downloader some is enough. (chequers is CPO admin so I guess you're ok for that one?)


braveplatypus' morgue already gave me a lot of ToDo.
I will try online morgues after that, but thanks for the advice !

Re: DCSS stat utility

PostPosted: Wednesday, 4th October 2017, 18:36
by lepoulpe303
braveplatypus wrote:I just realized, that first folder is only the games i played on one of the servers, since games from different online servers are saved separately, here is my other online morgue folder.

Ready for more headache?

This server saves the files in the offline style :lol:

On the other hand maybe that means less errors.

If you end up not being able to match the morgues or cant get data from the other save style don't worry about it.

Also, some of these games are from much much older versions, and some are from fresh versions, that might be the cause of some errors. Feel free to clip out super old games if that helps.


Drum roll ...
https://gist.github.com/oschwab/a9d27253a02a0ccec0cb7dd5c8751a46

I merged both morgues you gave me.

There are still some abnormal things (like the fact you died a couple of times in dungeon n/a:n/a ) and presentation got to be improved, but that's quite usable I think ...

Re: DCSS stat utility

PostPosted: Wednesday, 4th October 2017, 20:30
by braveplatypus
Two things i notice right away

It doesn't seem to be able to tell the difference between "Escaped" and "Escaped with the orb"

Monsters and some of their effects are separate in the kill listings (adder's poison) (jelly's acid) (large rock thrown by a cyclops)

Re: DCSS stat utility

PostPosted: Wednesday, 4th October 2017, 20:45
by lepoulpe303
braveplatypus wrote:Two things i notice right away

It doesn't seem to be able to tell the difference between "Escaped" and "Escaped with the orb"

Monsters and some of their effects are separate in the kill listings (adder's poison) (jelly's acid) (large rock thrown by a cyclops)


- Escaped vs Escaped with the Orb is in my todo list
- Monster & effects => bug reported https://github.com/oschwab/dcss_stats/issues/2

I don't know what to with quit (I honestly even did not know that you can quit a game)

Re: DCSS stat utility

PostPosted: Thursday, 5th October 2017, 11:15
by lepoulpe303
New report :
https://gist.github.com/oschwab/e1cf0a4d28d8ee56bf42056c1b9a5ca6

Escaped from dungeon (n/a:n/a) not listed in Dungeon stats anymore
Monsters effects ignored (adder's poison => adder)

Re: DCSS stat utility

PostPosted: Tuesday, 10th October 2017, 14:48
by lepoulpe303
New report :

https://gist.github.com/oschwab/7884cac ... 20e4075385

(+) Escaped with orb stat