Immortal Warwalrus report

Immortal Warwalrus was a team on the Usability Testing course. They did usability testing on DCSS caster tutorial in 0.6 development builds (near release) in spring of 2010. These tests were done in parallel to the UP team work.

The below is parts of the results report, which was mostly in Finnish. The hopefully relevant parts were haphazardly translated by evktalo.

Summary

We found that the tutorial has too much text and it distracted learning curve of players. Keyboard shortcuts were hard to learn and remember.

Players couldn't use mouse as much they wanted. Game forces users to use keyboard for many tasks and this distracts the flow of game play when they try to remember what tasks could be done with mouse and what tasks needs to use solely keyboard or mouse + keyboard.

Casting spells is made harder than it should be. This was clear when players first time tried to cast spells and remember the task.

Level up was too hard to notice and this made some users to think the game has crashed. Level up should be a good and clear thing that gives user the feeling of progress and success. By showing more clearly the level up process user feels more satisfied with the process.

Users had their left hand near WASD that is now days used for movement in almost every game, even rpgs.

Overview of testing

The testing aimed to assess the overall usability and playability of the game, and to find problems and/or bugs. The target of the testing was the caster tutorial (now hints mode). The aim was to gather information if new players can comprehend the commands in the game and specifically how well the mouse controls work (if the players choose to use the mouse). The testing is exploratory/affirming in nature, as it is difficult to give the testers specific tasks because of the random nature of the game. The usability attributes examined are learnability, rememberability and user satisfaction.

Because the game has many (keyboard) commands, the tests try to examine if remembering these commands is easy or hard, and what could be done to improve the situation. Also it is examined if learning to use the commands is easy, and is the user happy with the playing experience.

Before testing with testers, a heuristic evaluation was performed by a team member to identify problem areas before the tests.

Finding suitable users for the test

The desired testers were gamers without previous experience in roguelike games. We tried to find the best possible users for the test based on requirements from development team. This was done by asking possible users their gaming habits and past experience so that we could get the best possible results from tests. We found 3 different kind of testers that really helped us to test the technology from three different perspective.

Executing tests in test environment

Tests were done in controlled test environment where we could monitor the user and his actions. This was done with multiple cameras and microphones where no action was left unnoticed. Users were supported by moderator where moderator works as help when users really need it. We tried to keep moderator involvement as little as possible so that users had to rely on in game help as much as possible.

Analyzing the results

Videos were thoughtfully transcripted and analyzed and then written down on paper with as much details as possible. This helps the developers easily to see and find the problems in the technology.

Results

We found several problems with the technology. There were many severe problems that must bee addressed as soon as possible. In other hand technology had lot of already working properties. User experience ranged from possible new technology user to user that never ever wants to see technology again. This helped us to address many different problems from many sides of users that we think is vital to developers when they analyze the results and improve the technology.

Heuristic evaluation results

Heuristic Evaluation Suggestion for improvement
The story should be consistent, yet with multiple possibilities. Game has an overarching story that can be completed in multiple ways. Introduce the final goal in the beginning and keep the player aware of it.
The story and the world are interesting Games story and the world are quite similar to others in the genre, but include unique elements. There are still various possibilities to increase the amount of uniqueness.
Make Your Game Familiar, Yet Different. Compared to other roguelikes the game is familiar, maybe even too familiar. There are still various possibilities to increase the amount of uniqueness.
The goals are clear troughout play. The final goal is not clear to the player and the meaning/mission of different sub-dungeons are unclear (lair, mines etc.). Provide reminders of the final goal every now and then, keep the player aware of what (s)he has to do at all times.
Provide consistency between the game elements and the overarching setting and story to suspend disbelief. Consistency in game is good, great even. No Suggestion.
Player is interested in the characters because (1) they are like me; (2) they are interesting to me, (3) the characters develop as action occurs. Game provides several possible combinations of character specifications, propably one of everyone. Characters develop as the game goes on. No Suggestion.
Pace the game to apply pressure but not to frustrate the player. Vary the difficulty level so that the player has greater challenges as they develop mastery. Easy to learn, hard to master. Pacing the game is mostly in the players hands (resting in safety etc.). The randomness of events/monsters may frustrate the player (difficult monsters early). Enemies get harder as the game goes on and different enemies require different tactics. No Suggestion.
The game should give rewards that immerse the player more deeply in the game by increasing their capabilities (power-up), and expanding their ability to customize. It is possible to receive gifts from gods or find ancient items that are better than regular items. No Suggestion.
Players should not experience being penalized repetitively for the same failure. Wearing/using cursed items may frustrate the player if this occurs constantly. Potions or scrolls with negative effects may cause frustration too. Increase the amount of identify scrolls and detect curse scrolls in the very beginning.
Game should react in a consistent, challenging and exciting way to the player’s actions. Actions have very consistent reactions. Unidentified items give challenge and excitement, so do unique monsters. Possibly increase the amount of items that produce random reactions (scroll of random effects).
Player should be given controls that are basic enough to learn quickly yet expandable for advanced options. Mouse control easens the learning curve and key commands increase speed, but getting used to the controls and learning them properly takes time. Add an “Control tutorial” part to tutorial.
Do not expect the player to read the manual. New players (to the genre) propably need more guidance than the tutorial offers at the moment.. Increase the amount of teaching different things (items, monsters, actions) to the tutorial.
The first player action is painfully obvious and should result in immediate positive feedback. It is possible to die on the first enemy in the tutorial, but the beginning is explained well. Weaken the first few monsters to give the player more positive start. Tell the player when the monsters get tougher.
There is an interesting and absorbing tutorial that mimics the game play. There is a tutorial, but doesn't mimic the gameplay, it is playing the game with longer descriptions. Separate the tutorial to a independent minigame.
The game is enjoyable to replay. Game is hard to complete, so replaying is a hard issue to talk about. Game gets harder with every time player dies (ghosts). Game itself doesn't suffer from this, but the players opinion may. Are the ghosts compulsory, or could the player be given an option wether to include them?
The player experiences the user interface as consistent (in control, colour, typography, and dialog design) but the game play is varied. User interface is consistent for most parts, but there are some colour issues that may cause confusion (green color of poisoned corpses etc.). Usable items are colored by their properties which is good. Check the colours to match common opinions about colours (positive colours [yellow, green, white] for positive properties and negative [black, red, gray] for negative properties).
The interface should be as nonintrusive to the player as possible. It is easy to miss click and miss clicks can cause unexpected events that may not be easily recovered from. Border the clickable areas clearly to avoid miss clicks.
Make effects of Artificial Intelligence (AI) clearly visible to the player by ensuring they are consistent with the player’s reasonable expectations of the AI actor. The AI works good in the game. No Suggestion.
Controls should be intuitive, and mapped in a natural way; they should be customizable and default to industry standards. Controls differ a little from other similar titles, but are intuitive and natural enough to be learnable. Customization is available through macros. No Suggestion.
Game can be turned off anytime and continued from the point the player was in before quitting. Closing the game will erase the character if it's not saved by the player. Saving the character as default before closing the game?
Player should be able to save the game anytime and in any situation (as long it serves the context and the story). Saving is possible at all (reasonable) times. No Suggestion.
The player should always be able to identify their score/status and goal in the game. User interface has the players situation clearly visible. Goals are not known at all times. Keep the current goal visible/available at all times.
Player’s should perceive a sense of control and impact on to the game world. The game world reacts to the player and remembers their passage through it. Changes the player makes in the game world are persistent and noticeable if they back-track to where they’ve been before. Effects to the world are persistent (effects to walls for example (digging)) and dropped items are usually back-trackable. No Suggestion.
Challenges are positive game experiences, rather than negative experiences (results in their wanting to play more, rather than quitting). Sometimes challenges may become too early and be overpowered, this can frustrate the player. Challenges usually positive. Balancing the challenges based on players level (Weaken/toughen unique monsters based on level for example).
Game play should be balanced with multiple ways to win. The game can (apparently) be completed with all possible character combinations, but some are a lot easier than others. Balancing the world based on the character (different monsters for different characters for example).
Player experiences fairness of outcomes. Outcomes are generally fair, but sometimes the player may be punished for her/his weakness (priests for example).. Balancing the world based on the character (different monsters for different characters for example).
The game transports the player into a level of personal involvement emotionally (e.g., scare, threat, thrill, reward, punishment) and viscerally (e.g., sounds of environment). One can involve in the game emotionally, but detecting a threat for example may require further inspectation of the enemy. Make harder enemies look harder. Possibly add an "difficulty aura" or border to describe threat level.
The Player has a sense of control over their character and is able to use tactics and strategy. The character is very strongly under the players control. Different characters require different tactics and strategies, but combining different tactics is hard. ”dual-class” -characters?

User Test 1 Overview

Note: the report also included a full log of things that happened and a short interview after the game. These are not translated here. Below is a summary of observations and recommendations based on the first user test. The translation has been “streamlined” a bit - i.e. I didn't translate exactly all of it, but what I felt were the relevant bits. — evktalo 2011-01-16 09:45

Casting should be easier. Even though the tester ultimately managed to cast spells correctly, and considered it easy at the end, it was clear at the beginning that casting spells wasn't as easy and self-explanatory as it should be. The tester tried to cast spells by using the book from the inventory, and never found the menu which displayed all the memorized spells. Suggestion: instead of abcdef.. shorthands for casting, use 1234567890qwertyuiop… This is because this is the modern convention (in MMORPGs etc).

The tester didn't find out how the character can rest. When the character was on low health for the first time, the help text suggested to run away from enemies, and did not instruct how to rest. When resting came up in the help text, the tester was already immune to the pink text and ignored it entirely. It might be better to bring up resting when HP/MP was < 75%, or right at the beginning of the game when it is certain that the user will read the message.

It became apparent that there was way too much text in the tutorial. It is understandable that the devs want to include lore in their game, but in the instruction phase it should be toned down. The user soon became immune to all the instruction texts and often remarked, that the abundance of help texts makes playing harder and makes learning take longer. This was also mentioned in the final interview. During the test, it was noticeable that the tester only read the first couple of lines from the help text, then considered it to include little relevant information, and then skipped the rest of it, including the bits in the end that actually taught the player relevant information about the game. This further confirms the notion that there is too much text in the tutorial. It would probably be the best to cut much of text from the tutorial, or to completely remove the lore and teach the player, in short and information rich episodes, how the game is played and how each thing functions, so that the player will consider each help text useful and read all of them when they come up.

It was not apparent to the tester that the tutorial is actually the whole game with advice included. It would be good to relabel the tutorial to “beginner's help” or similar, that tells the player that it's the whole game with advice included, or to make a separate minigame out of the tutorial.

Item usage was unclear in many ways. Usage of potions was in the beginning unclear, and their effect likely remained a complete mystery to the tester. The same happened with weapons and armour, to which the user didn't pay much attention. Weapons and armour do not give any direct information what they do to the character when used, and you can't see from the character's stats what they and other items do to the character. Rings are of course marked in some way, but here I mean more about “armour, attack, defence, dodge, speed” etc. attributes, that are clearly visible when you observe the character, so that you can be certain if the new item is better than the old one and how it affects the character.

I think the above is somewhat confused, as AC, EV etc are visible on the screen. Also, the identification aspect might be confusing, and resistances etc are in the '%' screen, or '@' display, which might be unknown to the writer (and very likely to the tester). But many of the attributes are quite invisible, especially when it comes to weapons. This is an excellent proposal. We should have a tutorial about all the different stat screens.. — evktalo 2011-01-16 09:45

Using the mouse should be more streamlined, and more functionality should be accessible with just the mouse. Frequently during the test it was not clear to the tester if the action could be performed with just the mouse, or if the keyboard was required as well. As an example, when using a potion, the tester did bring the cursor over the potion, but then pressed 'q' from the keyboard instead of clicking it with the cursor. Another example was changing floors, where the tester tried many times to take the stairs by double-clicking. The mouse is a good addition to the game, and using it certainly helps playing, but the implementation is still incomplete and it should get more attention, so that mouse-usage becomes a full feature of the game.

Keyboard shortcuts during the game were often a big obstacle, and in the final interview the tester mentioned the shortcuts and learning them to be a problem. Same actions could be done with several different keys, and achieved through several different paths, which makes learning more challenging and remembering harder. When there are a million of different shortcuts, it is impossible to imagine that you could learn all of them during a tutorial, or even until you have played really long without help. Suggestion: a context-sensitive use command, which was aware of the environment and would do the action that was appropriate. If you were next to a closed door, it would open it, and next to and open door, it would close it. If you were on top of the stairs, it would take them. If there are several usable objects around, the game would confirm which one should be used and so on. This way, the shortcuts would be cut to a minimum and the user-gameworld interaction would be maximized. It is understandable that this would be a big change, but I would believe it would give benefits to both new and old users.

Shortcuts could include confirmations that differentiate the situations. In places, you are asked for Y/N, others y/n and even yes/no. In the test, this caused problems often, when 'y' didn't work, and SHIFT+y was required. In general, many of the shortcuts require shift+key, and the tester often interpreted 'Y' to mean 'y' instead of shift+y.

During the testing, the tester often ignored melee completely, and at times did not even recognize it happening. On the other hand, casting spells gives a certain level of graphical feedback, such as the trail of the spell, so it is curious that melee doesn't gie any similar graphical feedback. It might be good to have some sort of an animation, or other graphic notification, that the player is in melee with the enemies.

The character's status during the test was unclear to the tester. For instance, the fact that the character got hungry was apparent to the tester only after the moderator commented about it to the tester. Similarly, when the character quaffed a speed potion, it's effect was unclear. After discussing this with the developer (evktalo), it was concluded that the sides of the screen might be a good way to indicate the character status. As an example, after getting poisoned, “Poison” would read where it currently reads, but the sides of the viewport would have a transparent green overlay. When “speed” status was added to this, again “Speed” would display as currently, but two of the viewport sides would be green and two would be bluish. This would signify to the player that something has happened to the character.

As mentioned before, information should be brought out more. For instance, how the items affect character. As an example, what are the normal accuracy, damage for the character, what on earth is base attack delay, and what is affected by it? What does the weight of the weapon do to the character, and which armours make spellcasting harder? Where do I see all my items and their complete effect on the character? All these were unclear during the test, and it would be good to have information as important as this better presented to the player.

During the testing, the tester also did not notice gaining a level. The first time, the tester thought the game had crashed, when everything froze and no button functioned. Level-ups should be signified better, so that the player knows how to respond.

Also, the colour schemes of the game are not consistent. For instance, the tester did not notice that memorizing a spell failed, because both failure and success are in the same gray colour. It would be good to have consistency in the message colouring, so that good things (learning a spell) would be in green, and bad things (memorization failure, dying, quitting) would be in red. Following a commonly understood colouring scheme it would be clearer to the player if a thing that happens in the game is good or bad.

Tester's routines

The routines of the tester included keeping a hand near 'wasd' keys and using mouse for movement. Movement during the test seemed easy and fast. The tester used shortcuts for spellcasting after finding them, but aimed the spells with the mouse. You could count the repeated double-clicking of stairs as routines, as well as the tendency to want to take the stairs up on the first floor, even though it ended the game. It is suggested to change the first floor upstair textures to something that informs to the player clearly, that taking them will end the game.

User Test 2 Overview

The user had not played roguelikes before, but knowledge about CRPGs in general helped to settle into the game. The tester had the modern PC gaming hand position, that is right hand on the mouse, and left hand on the keyboard (wasd). At the beginning of the test, the hands hadn't found their place, but they wandered a bit - for instance, the tester used their right hand on the keyboard occasionally.

This tester was very cautious, and read every bit of instruction very carefully. Afterwards, the tester followed out the instructions in the best manner they saw. Because of this, the tester's playing time was shorter than the others. At the same time the tester completed all their tasks successfully. Not many things were left unclear during the testing. The biggest problems faced were understanding the Memorize-command initially, eating corpses, some of the keyboard shortcuts (remembering them), and a mild annoyance about the amount of the help texts.

The tester was from the beginning interested in the features of the game and was keen to experiment and find out what was possible. Here was where the Memorize-command came up. The tester wondered what the command did, because the help text did not explain it, and on top of the new spells there was just the text “can't memorize this spell now”. This confused the tester, until they got another experience level.

During the game the character got hungry. The help text instructed that corpses can be eaten, but did not tell how to proceed with that (or the tester didn't find out). Eating the corpses came up only later during the test anyhow, and the tester thought that later on the issue would become clear.

In the end, the tester remarked that the keyboard shortcuts were a bit messy, and remembering them was difficult. The tester suggested that some of the shortcuts could be in e.g. the number buttons.

The keyboard short cuts are in a way logical, because they usually are on a button that has the same letter as the first letter of the command. On the other hand this has led to the situation where all the shortcuts are all around the keyboard. Because of this, it would be better that the shortcuts were centralized in somewhere on the keyboard, such as the WASD area/half of the keyboard, so that the hands didn't need to wander around the keyboard. Additionally, it should be made clear if the movement is to be done with the mouse or the keyboard. Right now you can do many things with both, which clearly confused the tester and made learning difficult. It would help, if movement was restricted to keyboard and mouse was limited to e.g. aiming, inventory use and item examination.

Also, confirming things (yes/no), there are three different ways you're required to input your answers. The best solution would be to only have one, and additionally, they should be mouse operatable. A clear pop-up style query with YES/NO buttons would be good. Also, some of the keyboard shortcuts could be replaced with a situation-adjusting “use” command.

During the testing it was obvious how much tutorial and lore texts there were. You could see when the tester started to get frustrated with the amount of text. Because it is the tutorial, the amount of lore could be reduced, or removed completely, and just teach how the game playes. In the test it took the tester over four minutes to start playing, because they read all the text. Thus the lore could be removed from the tutorial, and only introduced when the proper playing starts. Also, some of the help could be simplified, with the principle of “what, why and how it works”. Now some of the help was explained well and the tester understood it, but from others there was a feeling of “why is there such a thing.” One of the things that was never clear for the tester were the stats of some ites, such as the damage or speed of the weapons. Now their stats were in percents (?). It would be better to use more straightforward values like bigger is better. The tester didn't find out if a bigger percentage was better than lower.

Recommendations

Short term recommendations

Keyboard shortcuts should be quickly adjusted. From the testing, it was clear that the shortcuts were too complicated and hard to learn. For instance, the spellcasting would work better with z → 1234567890qwertyuiop.

Also, implementing a single “use” command came up a lot during the testing as a good way to lessen the need to remember the several shortcuts. The yes/no confirmations should also be consistent, so that the player didn't need to guess what way the confirmation should be executed.

The mouse support should be more complete. There are currently a lot of actions that cannot be done with the mouse. Things like yes/no prompts, taking stairs, and interaction with items such as picking up, using and checking their properties, both in the game world and in the menus.

I'm a bit confused about the item interaction - I thought that was particularly handy with the mouse. — evktalo 2011-01-16 13:16

It would be good to have a consistent colour scheme, so that the player can easily make out if a thing is a good or a bad thing. On example is the learning of spells, where both failure and success give a gray message. This is repeated in many other choices and events where the text is grey, and there is no visual clue about the nature of the event.

It would be good to bring out the statistics a bit more, e.g. the case “what on earth is base delay”. It would be good to bring out the effect of armour, weapons and other items to the character's statistics, so that it'd be easier to tell if the item is better or worse than the preious item. For example, how do the heavier armours affect spellcasting and player speed? When there are a lot of things in the game that affect the character's qualities, they should be brought out clearly so that the player can react to them.

The amount of text in the tutorial should be reduced, and the tutorial texts should focus on relevant, game-learning information. Probably the best model for this would be “what/why/how”, so that the player would get the needed information quickly, and also be assured that all the texts are informational and good to read. It was obvious that not all the testers didn't have the stamina/patience to read through all of the texts (or to read them at all)

The tutorial should also have some sort of a information package about what the game is about, what's the purpose of the player character and what style of the game it is, so that the player has an impression what they should do and what sort of a game to expect.

I think that's what the website etc. should do. — evktalo 2011-01-16 13:16

The upstairs on the first floor should be changed so that the player instantly knows they're different than the other stairs, and they should signify visually to the player that the game will end when they take them.

Long term recommendations

There should be attention to the graphics in the long run. For instance, new animations in battle, and other interactions would be desirable. Also, the textures should be paid attention to, for example the look of the monsters and static objects are sometimes close together, so that the monsters are not easy to spot in the graphics.

In the long run it would be good to make the tutorial to a separate game, which teaches the basics of playing, and the most important features. This makes it possible that the player is left with the least amount of uncertainty about the game. The tutorial should be structured so that the game instructs the player through it step by step, telling what the player should do and how.

Logged in as: Anonymous (VIEWER)
dcss/usability_project/immortal_warwalrus_results.txt · Last modified: 2011-12-22 17:24 by XuaXua
 
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki