Item and monster generation stats

[gammafunk]

Guys it's gammafunk here to whisk you all away to a Wonderland of Numbers.

I made a mode for crawl to generate monster and item statistics. We didn't have a good way to summarize most of these statistics previously; hangedman's monster probability spreadsheet was very helpful, but couldn't account for vault placement, and we didn't really have item statistics at all.

The'res a public folder with Google spreadsheets giving stats averaged over many iterations for different versions. See the README for the most up-to-date description of the data and details on how to run objstat yourself.

If anyone notices any inaccuracies in the data, that info is welcome. Here are some statistics I'll possibly add at some point:

* An artefact property summary table. Summarizing this by equipment type and level would be too much, so maybe just a row for each property across items and levels.
* Spell and spell school summary.
* Summarize spells/schools for artefact books similarly to how I do brands on weapons/armour/missiles.

Suggestions for other stats are welcome, but keep in mind you can easily do further summaries in the spreadsheet itself.

[archaeo]

a Wonderland of Numbers

I prefer Mathmagic Land, thanks.

But yes, good show gammafunk, this is extremely useful and helpful, etc.

[Cheibrodos]

You're climbing up the rankings again after that last plummet, Gamma!

Really, nice work though.

[njvack]

This is awesome; thanks!

One thing that stands out now is that if you're looking for, say, a robe of Archmagi, you're going to be trying on something like 50 glowing robes. I kinda knew that before, but now the numbers are out there too.

[and into]

Aaaaaand—Stickied.

I imagine this would be of interest to many people making serious proposals in GDD. So now it's got a spot at the top.

[gammafunk]

I've fixed a bunch of bugs in objstat and rerun the stats. The google spreadsheet link has been updated in my original post, so see the new URL to get the most accurate data we have right now. This version now includes the stats for Abyss,Pan,Shoals,Spider as separate sheets but those are still not included in the AllLevels summary so that the latter can represent a "typical dungeon". Stats for Zig are available in a separate doc, since that has a lot of columns google docs doesn't want to merge the two documents.

The objstat command is in trunk now, so now people who can compile crawl can run this themselves. I'll look into setting up some kind of cron job to regularly run both -objstat and -mapstat and make an archive of the resulting text files.

[XuaXua]

Maybe it's just me and maybe I'm doing it wrong, but I think you need to at least allow the column widths to be adjustable so we can read the details of each cell.

[gammafunk]

I've widened the first column and frozen it in all the sheets so at least that is easily readable. Aside from that, there's not much I can do without making it writable, which would be a bad idea.

You can download the spreadsheet in a bunch of standard formats from File->Download As, so I recommend that if you're having trouble viewing certain fields. You can also make a copy in your own google account under File->make a copy if you're signed into google.

[Arrhythmia]

Okay, now add a column for the standard deviation, I'm curious.

[gammafunk]

There have been a few important bugfixes to objstat to prevent it from incorrectly counting certain monsters and items, from contents of levels that ultimately were rejected by the level builder. I've also added min, max, and standard deviation columns for the basic number fields.

So I've updated and reorganized the objstat google spreadsheets; they're now in a public folder you can find here, which should be a permanent URL, as opposed to the files which change URLs every time I recreate them. The original post is edited with the details on what the files contain, but the short summary is there's an AllDungeon (or 15-rune) fileset and a 3-Rune file. Thanks to neil for the min/max/SD suggestion and for generally being awesome and helpful unlike some half-baked fire giants out there who just generally waste everyone's precious aut.

[njvack]

Actually: the mean number of an item in a game isn't the most useful stat. Quantiles (say, min, 1st quartile, median, 3rd quartile, max) would be better. Mean & SD are only informative if the data are likely to be normally distributed; these are probably not.

In essence, I'm more likely to want to know "what is the number of item X I'll find in a typical game?" than "if I play 100 games, how many total item X will I see?"

[gammafunk]

Actually: the mean number of an item in a game isn't the most useful stat. Quantiles (say, min, 1st quartile, median, 3rd quartile, max) would be better. Mean & SD are only informative if the data are likely to be normally distributed; these are probably not.

In essence, I'm more likely to want to know "what is the number of item X I'll find in a typical game?" than "if I play 100 games, how many total item X will I see?"

The mean and SD are informative for any distribution of data where the mean and variance are defined, which they certainly would be for this data since we have a finite range of values (there are maximums on the numbers of items and monsters). Just because the mean and variance are the parameters for a normal distribution, that doesn't mean you only consider the mean and variance if you have normal data.

Furthermore across these independent iterations, these data would be reasonably approximated by a normal for a large N like 100 for a very large majority of item and monster types. The min and max are there for those that might always generate at some minimum or maximum number. So the best answer to the question of "what is the number of item X I'll find in a typical game?" is "the mean number generated across iterations", which are various the "Num" fields. It's never bad to have more statistics if calculating them is free, but we have to watch the size of output we generate as well as the lines of code to maintain (and there's a lot for objstat already).

The statistics I'm interested in adding are accounting for things we don't track at all, like artefact properties (how often does e.g. +Int occur at various values) as well as spells and spell schools across the ordinary and artefact books. These are a bit trickier to represent (something like how I do the brands might work), but I think also a fairly low priority (and I've worked on this enough for the time being).

[njvack]

Oh, I'm being silly. If there's an unvarying X% chance of an item being generated on a given level, the count should follow a poisson distribution. The mean == variance in a poisson distribution, and quick eyeballing suggests that this is mostly closely true of things that I'd expect in the data (consumables, jewelry, not monster gear). So, printing both mean and variance would let you eyeball how much items deviate from an "expected" poisson dist.

It might be interesting to get histograms for some important items (?enchant, ?recharging, etc) and see what the dist actually looks like. I'm not saying this is something you should do! But I might try it, because I am curious

Also, I don't mean to imply that wild swings in itemgen are bad; "that one game where I only found four ?EA scrolls" is interesting, as is the 7-!benemut game.

[gammafunk]

Oh, I'm being silly. If there's an unvarying X% chance of an item being generated on a given level, the count should follow a poisson distribution.

The count isn't poisson, one obvious reason why being that a poisson has infinite support, which doesn't make sense in a game with a finite number of items generated per level. There's a discrete uniform on the number of level items (usually 3 + 3d11, but there are rare rich item levels), a discrete (non-uniform) on the item base type, and then a discrete (non-uniform) on subtype. If you want a simple distribution to think about as a model, you could think binomial with e.g. N = 21 and some unknown p for the item or monster.

The mean == variance in a poisson distribution, and quick eyeballing suggests that this is mostly closely true of things that I'd expect in the data (consumables, jewelry, not monster gear).

It isn't mostly true of consumables: all food, most potions, and quite a number of scrolls deviate a lot from mean = variance. You have to square the SD to get the variance, in case you forgot. Note how mean != variance in nearly all across-base-item-type totals (e.g. All Jewellery); magical staves are one type were mean = variance, but this is a quite rare item type. A binomial has a mean of np and variance of np(1-p), so using this distribution as a model, the mean and variance are the same the same as p becomes quite small, but generally this is also a poor model for the data as we tend to see variance larger than the mean.

So, printing both mean and variance would let you eyeball how much items deviate from an "expected" poisson dist.

The reason we're including the mean and the variance isn't because we care to detect deviation from some distribution, it's because we're most interested in just estimating the mean for balance purposes. The sample mean will follow a normal for a large enough N (by CLT), so we can estimate the mean and know how accurate our measurement is using the variance. The min and max are important for the basic questions of how few of X or how many of X could a player see. Histograms and tables of probabilities of seeing N of X in a branch or dungeon would be nice but that's generally too much detail for making design decisions.

[njvack]

I guess where I was going with the "SD isn't very informative" comment was for items like ?HW, where a mean of 3.64 and SD of 2.13 mean you should expect a negative number of scrolls in 4% of games, which doesn't happen. It does actually tell you stuff , but it's not super simple.

You're quite right on the support of the poisson distribution... at the same time, the mean ~= variance actually does hold for a bunch of consumables; the ratio of mean/variance is around .9 for quite a lot of them (and I'd generally expect it to not be 1, because shops and vaults and such). It's about .65-.7 for things like ?enchant, which makes me think maybe those get generated as specific treasure a bit more often. But that doesn't hold for, say, !benemut, so I'm actually probably cherry-picking.

Thanks for the suggestions on how to think about itemgen distributions.

One thing I imagined this tool being used for was to catch the occasional vault that generates way to many of a particular item, before the vault lands in stable. But I'm no dev, so yeah.

My other thought was people mentioning that they wind up with a bunch of tactical consumables at the end of a win, but I realized there may well be confirmation bias there: games that generate a lot of items will tend to have generated lots of items early, which will mean they're easier games and more likely to be won.

Anyhow, I realize I'm kinda just wanking here, so, um, thanks for having this discussion with me