A few days ago on IRC, blackcustard asked me to do a blog post on a particular save-compatibility fix I made about a month ago, because it’s an interesting (I prefer “dirty”) kludge. I should start, however, with an introduction to how save compatibility works in Crawl.

Version tags

For save compatibility purposes, the actual version number (for example, 0.12.2 or 0.13-a0-2578-ged92a99) makes almost no difference. Instead, there are three version numbers (which we call “tags” for historic reasons) associated with each save and with each version of crawl: the character format, major version, and minor version. In current trunk these are 0, 34, and 44.

  • The character format tag is stored in the “character info” section of the save: the basic information such as species and level that is displayed in the save browser. If this section is changed in an incompatible way, the character format version would be incremented. However, Crawl would not even see the old saves; this would be rather bad (they might accidentally be overwritten, for example), so we have never made such a change: the format is still version 0. Appending new information to this section requires only incrementing the minor tag (see below), which is much safer.

  • The major version tag is stored in each section of the save (each level, character, transiting monsters, and so on). Changing this version indicates that Crawl is no longer compatible with the old save: the save will be coloured red in the save browser and cannot be loaded. The major tag is never incremented within a release (e.g. between 0.12.0 and 0.12.1), only in trunk. We try to avoid doing this when we can, because it means trunk players will be unable to transfer their saves to newer versions to get bug fixes; it furthermore means that local players will have to keep the previous stable release installed until they have finished their games.

    The major tag has been incremented 34 times, but the rate has slowed down significantly. It was incremented several times in 0.8 development, once in 0.11 development, and once early in 0.12 development. Through heroic effort, kilobyte was able to restore save compatibility between 0.11 and 0.12 despite the major tag change. Therefore, current trunk can load saves from 0.11 and 0.12 releases, all but the earliest 0.12 development versions, and all 0.13 development versions.

  • The minor version tag is also stored in each section of the save, and is used to indicate changes that require special handling for loading old saves. The save-loading code if full of checks that say “only do this if the save’s minor tag is greater than N” or “initialize this new field if the minor tag is less than N”. Older versions of Crawl will refuse to load a save with a too-new minor tag (that is, we do not provide forward compatibility between versions), but newer versions will see the old tag and apply the appropriate fixups on load (which makes the save incompatible with the old version of crawl).

    The minor tag is reset to zero whenever the major tag changes. A major bump provides an opportunity to clean up the old save-compatibility code, since the old saves won’t be loadable anyway; and to remove or rearrange enumeration values that could not change without breaking compatibility. The minor tag has been incremented 44 times since the last major tag (0.12-a0-109-ged95631, in August 2012).

The upshot of this is that, for the most part, online players can keep transferring their saves to the latest version of Crawl. In the rare event that we change the major version tag (once a year maybe), the servers keep those saves on the old version, so the players can continue even if they don’t get bug fixes.

Among the situations that require compatibility code (and hence incrementing the minor version tag) are new fields or data structures added to monsters: trying to load those from the old save would either cause corruption, or crash. Likewise, any change to enumerations like monster type, spell, dungeon feature, other than adding a new value at the end, requires at least a minor version bump and code to shift all the old values.

Bugs: the difference between theory and practice

All the rules about how to preserve save compatibility, and what does and doesn’t need special handling, can be kind of complex (see docs/develop/save_compatibility.txt). It’s no wonder, then, that occasionally there are problems. The development of Spectral Weapon introduced two save compatibility issues at different times:

  1. Spectral Weapon was written before elemental wellspring and polymoth, but was merged into trunk later. The merge, unfortunately, left spectral weapon’s monster enum in its old place, before the two “older” monsters. Thus an elemental wellspring from an old save could be loaded as a spectral weapon (probably resulting in a crash), or a polymoth could be loaded as an elemental wellspring (no crash, but a wellspring with incorrect statistics and no spells).
  2. Spectral weapons got a ghost_demon structure to store their statistics (AC, attacks, etc), which differ from one spectral weapon to the next. An old save would not have this structure, but the code would try to load it anyway, getting out of sync and causing a crash when some later part of the save was misread.

At some point later we started noticing these apparently-corrupt saves that would crash on load. After some investigation we found that they seemed to all have a “spectral weapon” on the incorrect level. That let us track down the second, newer bug. But without the version tags, how to detect and fix it?

Fixing the bugs

Kilobyte made a first pass at fixing the ghost_demon problem. For potentially affected versions (remember that because of the missing tag, this check caught some good versions as well), he used read-ahead to determine whether it looked like the saved SW had a ghost_demon structure. If it did not not, the weapon was replaced with a placeholder “ghost” monster.

However, after kilobyte made but did not commit his patch, we discovered the other problem: sometimes neither the player, nor any ghosts on the level, had the Spectral Weapon spell. More digging showed that the spectral weapon had Primal Wave and Summon Water Elementals, and led us to the enum problem. To fix that, we needed some way to tell which monster it was supposed to be. We noticed that all three of the affected monsters (SW, Wellspring, and Polymoth) had different speeds, which happens to be stored with the monster rather than being loaded from the monster class. Then qoala pointed out that the ghost_demon change to spectral weapon happened in the very same batch of commits as the one that changed its speed from 25 to 30. We could therefore kill two birds with one stone by using the monster speed to determine the true monster type, and didn’t need the look-ahead.

Ultimately, for saves from affected versions (between minor tags 38 and 39), for any monster claiming to be one of those three, we check the monster’s saved speed. If it is 10, it must be a wellspring; if 12 it must be a polymoth; if 30 it must be a correct spectral weapon; and if 25 it must be a spectral weapon with a missing ghost_demon structure (so we remove it, leaving a placeholder ghost behind). However, haste, slow, and friends affect the speeds stored with the monster; we therefore have to check for numbers like 15 (hasted wellspring), 16 or 17 (slowed old spectral weapon), and so on. Fortunately none of the three base speeds is 2/3 or 3/2 of any of the others, so we are still able to distinguish on the basis of speed alone.

For all the gory details, see commit 0.13-a0-2175-ga079a5c. One day we will break compatibility again and increment the major version; then this kludge can return to the ether from which it came. Until then, enjoy! :)