The Dungeon Olms Report

5 part tutorial - Usability test

Credits

Faculty of science
Department of information processing science
Usability testing-course

                                                      Teemu Kaukoranta
                                                      Miikka Kuutila
                                                      Sebastian Turpeinen
                                                      Perttu Piirainen
                                                      Tuomas Ristioja

Abstract

This report concludes the results of usability tests conducted for Dungeon Crawl Stone Soup, an open source roguelike game. Tests were done during the spring of 2011 as a part of the Usability Testing-course run by the Department of Information Processing Science of University of Oulu. During the test quite a few problems were found, varying from critical to not so. The test was planned and conducted by students of the aforementioned department, as per requested by the DCSS developer team.

Testing plan

The testing plan for this test was initially written in Finnish. Since the end report was requested to be in English and the initial testing plan had to be included, we have translated the plan too. This part is probably only interesting to teaching personnel, as it contains information which was relevant before the test (or if the test is to be repeated).

Background

The target of the usability tests was to test the tutorial mode of Dungeon Crawl Stone Soup (later DCSS). The aim is to find any possible flaws in the tutorial and to offer a solution to them. As per requested by the developer team, none of the test persons had any previous experience with roguelike games. The tests and reporting of the results form the output for the Usability Testing course. The tutorial mode in itself consists of 5 parts and is designed to replace the old tutorial of the game.

The term roguelike references to a certain type of turn-based computergame that uses tile graphics or more commonly character based (ASCII) graphics. The term roguelike comes from a computer game published in the 80’s called Rogue. Roguelikes have since grown a kind of a cult following as the somewhat randomly generated game content and die-and-lose-everything attitude make for a challenging but rewarding experience.

Goal

As mentioned before, the goal of this test is to find any flaws in the newly introduced 5-part tutorial of DCSS and offer solutions to them. Having a tutorial is very justified, as roguelike games typically require a very different mindset compared to contemporary roleplaying games, and they are also full of hotkeys and abilities which must be familiarized in order to be successful in the game.

The tutorial is divided into 5 parts which progressively get more complicated, at least to some extent.

  • Part 1: Movement and exploration
  • Part 2: Monsters and Combat
  • Part 3: Items and Inventory
  • Part 4: Magic and Spellcasting
  • Part 5: Gods and Divine Abilities

Even finishing the tutorials can be challenging for an inexperienced player, as the player’s character can die quickly if the player doesn’t pay attention. The tester was expected to finish all the five tutorials; approximately 30-50 minutes was allocated for this. However the tests revealed that the playing time was severely underestimated; for an inexperienced player the tutorial takes about 120 minutes to finish).

Thus the objective of the tutorial is to teach a player without any prior roguelike experience to play the game. The tutorial should be simple, educational and easy to pick up, as well as fun and interesting. After the tutorial the player should feel like he has learned to play the game, and wants to proceed to the actual game.

With the usability tests our team tried to analyze if the aforementioned attributes are really shipped with the tutorial; we wanted to find out if the tutorial feels understandable and playable for a novice. However, it’s also worth pointing out that the focus of the tests was only in the tutorial; the tester wasn’t offered the possibility to play the real game after completing the tutorials. This was purposefully excluded from the scope of the tests.

Planning the tasks and scenarios was easy, since the tutorial is already divided into 5 parts, which can only be finished in one way (as long as the player keeps moving forward and follows instructions). Thus the scenario was expressed to the player in one sentence: “Your task is to start the game and play finish all the five parts of the tutorial” as the test setting didn’t require anything more complex.

Test persons

The team chose 4 test persons to conduct the tests, thus - along with the pilot - a total of 5 tests were done on the system. All the test persons (aged from 19 to 27 years) were roughly from the same user goup: experienced gamers and computer users without prior roguelike experience. Conducting the usability tests with this kind of user group is based on the assumption that every tester should have at least some gaming background; testing the tutorial with players without prior experience in computer games would skew the results too much. This is because the tutorial mode in itself doesn’t aim to teach the player basic gaming skills and concepts (such as health bar or magic points), but it does aim to introduce roguelike concepts to the player.

Roles

The role of moderator was assigned to Teemu Kaukoranta; he’s responsibility was to familiarize and motivate the tester to the test setting and to interview the testers. The moderator observed all of the tests within the same room with the tester; if the tester happened to get stuck in the game, the moderator assisted him. However, it was of course desirable that the test person tried to solve the problem by himself.

The rest of the team acted up as loggers; they watched the tests via one-way mirror from the observing room. It was also their responsibility to ensure the recording of the tests. Also if any of the tests had happened to fail in an unanticipated way, the observer crew was obliged to help the moderator as needed.

Test environment

The tests were conducted in the usability test laboratory of the Department of Information Processing Science. The laboratory itself was divided into two rooms; one for the testing and the other for observational purposes. One-way mirror was included between the rooms to make the observing and logging as easy as possible. It was also ensured that the observational crew was in place before the moderator came in with the tester. After the test, the tester was offered a chance to take a brief look into the observation room also.

The game itself was executed on a fairly modern IBM Thinkpad laptop with screen resolution of 1280×1024 pixels. Also an external keyboard was hooked up to the laptop as the layout of the internal keyboard is too ‘compressed’ for smooth roguelike gaming experience.

The tests were recorded using two video cameras. One video camera was positioned at level with the test person. The main objective of this was to record all the physical reactions the tester expressed and to record the tester’s usage of the controls. Also the interviews was recorded using this camera. The other video camera was positioned in the ceiling recording solely the testers usage of controls. The 2nd video camera was mainly for backup purposes if the first camera should have had failed during the tests.

The gameplay screen was captured independently straight to disk. This was made to ensure that the gameplay and tester’s actions would be easier to analyze. However, as the analyzing sessions proved, the direct screen captures suffered from poor refresh rate and audio synchronization problems.

Interviews & conducting a test

Every test began with preparing the test room properly. This included setting up the cameras to record and beginning the capture. After this the moderator went to greet and guide the test person into the test laboratory (every tester was asked to arrive to the lobby of the department).

Each test session included two interviews: both before and after the test. In the pre-test interview the test person was asked about name, age, gaming experience and roguelike experience. The moderator introduced the task at hand to the test person, and tried to encourage the tester to be frank about his opinions and to speak and think out aloud as much as possible, in order to make the reviewing of the material easier. It was also emphasized that the team was in no way affiliated with the DCSS development team and the testing team was only interested about the tester’s personal opinion, not what he thinks others will think. The recording of the tests was also disclaimered to the tester: none of the testers seemed to be intimidated by the presence of cameras.

After the test the moderator tried to gauge what the test person thought about the tutorial. This interview was not structured compared to the pre-test interview, and it was very much up to the moderator to lead the interview and to get the deepest feelings of the tester without manipulating his answers in any way.

Some example questions from the post-test interview are:

  • What kind of feeling the tutorial left?
  • Did the tutorial feel boring?
  • Did any part of the tutorial feel especially difficult or annoying?
  • What elements of the game felt enjoyable?
  • Could you imagine to play the game voluntarily?

Thus one test can be expressed in steps as follows:

  1. Prepary the test laboratory; start recording
  2. Welcome the test person into the laboratory
  3. Motivate and introduce the purpose of the test for the tester
  4. Pre-test interview
  5. Start the test; elaborate the test scenario for the tester
  6. Execute the test with the following conditions:
    1. The moderator will help if the tester gets hopelessly stuck in the game. Otherwise the moderator remains silently and sits behind the tester (not too close to avoid too intensively observant feeling).
    2. Interrupt the test, if:
      1. the tester wants to quit
      2. the time limit is up (60-90 minutes)
      3. piece of equipment breaks rendering the test useless
      4. some other high-impact event occurs
  7. Post-test interview and ending of the test.

Expert reviews on the test material

This chapter contains a brief description of the preliminary evaluation of the game. Before the actual usability tests the team conducted a pilot test to detect possible points of failure, both gaming and test-set wise. Also each member of the team did a cognitive walkthrough to detect the most obvious problematic aspects in the game. The pilot test itself was enough to prove that playing the tutorial can be challenging experience for a new player; the game offers a great amount of information in a fast-paced tempo. This may lead to frustration and uncertainty when the player misses a single vital point of information. Thus some amount of patience and concentration is expected when playing the tutorial.

The information which is given to player is moderated by blue infotiles in the game; when the player steps on the infotile the game displays a chunk of information on the bottom left of the screen. This kind of solution is simple and functional in itself, but as couple of the cognitive walkthroughs prove, it might be useful to point out this fact in the beginning of the first tutorial in order to ensure the attentiveness of the player. Other improvement might be to adjust the functionality the infotile; currently the text is displayed only by the first time the player decides to step on the tile: displaying the text every time might help to ensure that the player receives the message. This, however, comes with the drawback of creating even more informational spam for the player to read.

The other thing worth mentioning is the difficulty of the tutorial; “The points in which the player may die are too numerous - in my opinion there should be none. Playing the tutorial isn’t surely very thrilling for anyone so its playing time should be minimized. Because the tutorial is designed for inexperienced players, is it wise to scare them away by killing them multiple times in the tutorial?” Also the slight repetitiveness was seen as annoying: “In several cases there were battles which really didn’t teach anything new. Fighting is simple, so there’s no need to harp it to the player.

Below are the listed minor discrepancies the team found before conducting the actual tests. Those flaws which occurred also in the actual usability tests are not listed separetely here. The list here represents just the thoughts of the testing team and should be reviewed as such.

  • The player is able to write the character’s name without selecting the focus for the name input textfield. Agreeably this kind of approach is fast but may lead to minor confusion.
  • There’s too much useless space between main menu and the description texts located in the bottom of the screen.
  • Completed tutorials are not marked as completed in the tutorial menu. Should the already passed tutorials be marked, at least per game?
  • When the player dies in the tutorial, should there be option to continue immediately? This, for example, might be something as simple as dialog asking “Do you want to try again?
  • The autopickup is described in the third tutorial; before this happening, a novice may wonder why some of the items are picked up automatically: “I didn’t find out the logic why some of the items appeared automatically in the inventory and some didn’t.
  • There could be a blank space between the hit points and the magic points. Currently it feels that the GUI doesn’t fit to the window.
  • According to the game, the autotravel doesn’t lead the player to dangerous areas; these should be marked with red X’s, but this isn’t the case.
  • In the combat tutorial the player easily dies in the with numerous rats. The info-tile of the room is also too easy to ignore!
  • I died again in the second level of the Items and Inventory -lesson. The tutorial seems to be very difficult. Do the enemies have a critical chance to hit or something? The second time the goblin didn’t do any damage, while at the first time a single hit was able to wipe half of my hitpoints away.
  • The text area is too small in some situations. For example, if new information is given to player, it’s too easily ignored.

Executing the pilot test

The pilot test worked well, excluding the fact that one team member forgot to bring his own videocamera so the test was captured only on one camera. We learned that the test will take closer to 60 minutes than we had at first anticipated.

The pilot test wasn’t very successful when it comes to result integrity. The test person (a team member) said he wasn’t able to relax or to concentrate, and it is clear he was in a hurry to finish the test. Because of this he missed a lot of vital information.

Executing the actual tests

The test were successful and apart from pilot the test situation and cameras didn't seem to bother the testers. All testers were male with some studies in department of information processing science. They all also fulfilled the set criteria. Everyone was active player with some experience on role-playing games but non on roguelike games.

Test situations

As said earlier pilot tester was a group member and even though test findings weren't very useful the test situation itself had no problems. Tester was frustrated with the game and ended test after 40 minutes. At this point it was an axiom that the estimated 60 minutes per test was enough.

First tester was 24 years old experienced on gamer who specially liked online role-playing games. Test had to be stopped by moderators since the laboratory was booked only for hour. At this time tester was playing the third lesson. Otherwise there was no problems on the test. Player was frustrated with multiple control possibilities and had troubles finding suitable combination. This shows on the video where tester uses arrows and numbed for moving, mouse for selecting and other keys for actions.

Second tester was 23 years old. He also was experiences gamer but didn't have that much experience on role-playing games. Test situation didn't seem to bother the tester since he was joking on the lessons. He also commented the experience well. Game style was pretty aggressive and he advanced to final lesson. Player relied more on testing things than reading the lesson notes. This time test session was made longer, about 80 minutes. Test session didn't have any problems.

Third tester was 19 years old first year student. He had similar gaming background than the first tester with lot of role-playing experience. Test took about 90 minutes and tester got to final lesson. Game style differed a lot from previous. This time tester was more calm, didn't comment that much and read the lesson notes always first. It seemed that the tester wasn’t very familiar with the “think out aloud” approach.

Last tester was 21 years old student from department of geography with some studies in information processing science. Gaming experience was mostly on RTS's but also some on the role-playing. Since in previous tests non of the testers were able to finish all the lessons we decided to start the last session from the third lesson. Tester was instructed to play lessons 1 and 2 before the session. Player was interviewed about these and said that didn't have any big problems and that the lessons were interesting. Tester was similar to the second tester. Seemed to be relaxed and enjoying the game. Apart from the other testers he used mostly just the mouse for controlling, keyboard only when must. Test took about 80 minutes and tester was able to finish the last lesson.

Capturing the tests on video

Test were recorded with two cameras and with screen capturing software. First camera was pointed to tester and the second to the keyboard. With the exception of the pilot test where no keyboard camera was used. Keyboard videos were restricted to 60 minutes since we were not able to connect the camera to computer and regular tape had to be used. Both cameras were positioned well and the videos showed what was planned. Screen capturing was handled with CamStudio software. Program wasn't the best possible since there was someone syncing problems with the audio and video. We didn't notice this in the pilot testing. Luckily the sync error wasn't too big and the videos were usable. For future test other programs should be considered.

Analysing and interpreting test material

Analysis

The materials we got from these tests were videos of the starting interview, the test itself and lastly from the interview after the test was done. Both the interviews and the test were captured by 2 different video cameras (one located in the ceiling, other in testers right so that he couldnt see it while testing the game) and by the screen capture program in test computer. All these video captures had sound source (ie. mic) so that syncing the materials would be easier when analyzing the tests. The best material to view these interviews were the videofiles from the camera to the right side of the tester, as you could also see testers face when he gave answers to questions.

After the tests were done, everyone from our group got one test to watch again and analyze. Each member of the group had all the materials collected from the test he was analyzing. Due to problems with different codecs, few of the members of our group reported the screen capture video lagging or unclear. Another problem that we faced was that sound in screen capture video would start lagging more as the video went on, and at the end of the video file audio could be few minutes behind the picture

After everyone had analyzed their test, we would meet up and watch some of the materials together. We tried to watch material which showed and highlighted some of the problems in the tutorial part of the game. After we had watched the materials, we discussed about the problems we saw and also what were the sources of these problems in usability. After these problems were discussed, we agreed what everyone would do for the end report and how we would be collecting all the problems we saw in the material. It was agreed that everyone would report the problems they saw in their material on google documents group, that the group created and also we would be using the template model used in DCSS wiki.

In this meeting we also agreed that the contentlog will be done from test 4 and the transcript would be done from the test 1. Both of these documents are included as appendices.

Interpretation

Observations on usability were done before the tests, during the test and after the test. Before the test every member of the group did a cognitive walktrough from the tutorial of the test and wrote a report from it. Before the tests we also discussed the things we found problematic during the walktrough. This gave us some thought on what to look on during the tests. Later some of the tests would confirm some of the walkthrough results.

During the tests members of the group who were observing the test would take notes from the problems they saw. This helped when we would be analyzing the materials afterwards. When analyzing the test from videos afterwards, the most important video to watch was the screen capture video. In this video you can see most of the problems concerning the tutorial. Videos from other sources were useful too, for example when player was having problems with controls you could see it better from the other cameras. Also a summary was made from the contents of starting and end interviews, and it can be found in this report at x.

Every problem found in individual analysis was collected to chapter four of this document. The problems were reported using template that is used in DCSS wiki. Before making this document final, the existence of these problems was discussed and evaluated together. Filling the template to shared location where every member of the group could modify them made it easier to know about the severity of the problem and also in what tests has it occurred in.

Most of the problems (if not all) we found were somehow related to the fact that the player was new to playing this game, but otherwise only way I could categorize them would be the heuristics they are violating. The findings mostly violate the heuristic 9, this happened most likely because we are testing a tutorial which is assumed to teach new players to play the game. Very few of these usability problems are a nuisance to players who have played this game longer and have already learned the controls and features of the game.

Findings and recommendations

The following heuristics are used to categorize the problems (Pinelle, Wong & Stach: Heuristic Evaluation for Games: Usability Principles for Video Game Design, 2008):

  1. Provide consistent responses to the user’s actions.
  2. Allow users to customize video and audio settings, difficulty and game speed.
  3. Provide predictable and reasonable behavior for computer controlled units.
  4. Provide unobstructed views that are appropriate for the user’s current actions
  5. Allow users to skip non-playable and frequently repeated content.
  6. Provide intuitive and customizable input mappings.
  7. Provide controls that are easy to manage, and that have an appropriate level of sensitivity and responsiveness.
  8. Provide users with information on game status.
  9. Provide instructions, training, and help.
  10. Provide visual representations that are easy to interpret and that minimize the need for micromanagement.

The problems are scaled from 0 to 4 to rate the severity (Nielsen):

  • 0 = I don't agree that this is a usability problem at all
  • 1 = Cosmetic problem only: need not be fixed unless extra time is available
  • 2 = Minor usability problem: fixing this should be given low priority
  • 3 = Major usability problem: important to fix, so should be given high priority
  • 4 = Usability catastrophe: imperative to fix this before product can be released

The following problems were identified during the tests:

Problem summary 1. Player does not realize his character has died.
Problem description When player plays the tutorial first time, he may not realize when his character dies. At least one of the testers asked if there was a bug in the game. This especially may happen when the player is not aware of the importance of the description texts in the lower left corner.
Screenshot(s)
Scenario(s) New player starts the “Lesson 2” tutorial and has not yet figured out the importance of description texts. While playing his character dies and he does not realize this as he may not read the battlelog first.
Breaks heuristic(s) 9. Does not provide adequate training and help.
10. Provide visual representations that are easy to interpret and that minimize the need for micromanagement.
Severity 3. Major usability problem
Occurred in Expert review, pilot, test one.
Suggestion(s) Add information tile explaining the importance of description texts/battlelog. Changing the character image and/or the “death text” color would help.
Related page(s)
Comments
Problem summary 2. Player does not learn to chop food
Problem description The information on how to chop food in tutorial “Lesson 2” is easy to miss. New player has just found out how to fight and needs to kill the rat before making food, so player in rush easily just runs trough the information tiles never reading the information.
Screenshot(s)
Scenario(s) New player has just learned to fight and destroys both training dummies and the rat. After this he picks up the sword, goes to next room, never reading the information on how to make food.
Breaks heuristic(s) 9. Does not provide adequate training and help.
Severity 3. Major usability problem
Occurred in Expert review, pilot, test one, test two.
Suggestion(s) The rat should already be a corpse when player comes to the room. Also information tiles on how to chop food should be before the corpse not after, so that player does not have to go backwards (this makes chopping food easier to learn).
Related page(s)
Comments
Problem summary 3. Player does not notice the ability to rest
Problem description The information on how to rest and heal is short and not very informative. It's very easy to miss.
Screenshot(s)
Scenario(s) The one line information shows after fight and is almost immediately lost since new information scrolls in.
Breaks heuristic(s) 9. Provide instructions, training, and help.
Severity 3. Major problem
Occurred in Expert review, pilot, test two, test one.
Suggestion(s) Add more information about the basic concept of turns to make player understand that not only one click heals fully. Possibly make the player to confirm that he/she has read the information.
Related page(s)
Comments
Problem summary 4. Player doesn’t learn how to bind equipment to hotkeys
Problem description The information is pretty long and hard to understand. Also seems that beginners don't value this feature very high.
Screenshot(s)
Scenario(s) Player reads the information, doesn't understand it fully or quickly tries to follow it. When failed just moves on since doesn't feel the need for this feature.
Breaks heuristic(s) 9. Provide instructions, training, and help.
Severity 1.
Occurred in Pilot, test one, test two, test four.
Suggestion(s) Possibly removing this from the lesson 3. Maybe moved into some advanced features lesson.
Related page(s)
Comments
Problem summary 5. Finding the weak walls
Problem description The information instructs to try to destroy walls with wand. Room has 4 special wall blocks that draws players attention. Player tries to destroy these wall and is confused when nothing happens.
Screenshot(s)
Scenario(s) Player read information and tries to destroy special walls with wand. Nothing happens. When informed that breakable wall is one of the normals player feels fooled.
Breaks heuristic(s) 9. Provide instructions, training, and help.
10. Provide visual representations that are easy to interpret and that minimize the need for micromanagement
Severity 2.
Occurred in Pilot, test two, test three, test four.
Suggestion(s) Even though this is not vital part of the lesson, player shouldn't be fooled by the lesson. Removing the special walls could help the player to search the walls better.
Related page(s)
Comments
Problem summary 6. Player does not understand what is the easiest target / descriptions of enemies
Problem description When new player has the choice of picking his fights, his information of the enemy is limited to descriptions. These descriptions themselves may be informative to players who have played the game longer, but new players had problems understanding them.
Screenshot(s)
Scenario(s) New player starts “Lesson 2: Monsters and Combat”, after killing some rats and training dummies, he comes to room with three doors and three different monsters. He reads the descriptions, but ultimately chooses the monster to attack at random, since he does not fully understand what he has read.
Breaks heuristic(s) 9. Does not provide adequate training and help.
Severity 3.
Occurred in Test one, pilot, expert review.
Suggestion(s) Add information tile to the tutorial explaining these descriptions. Maybe moving this to some advanced lessons tutorial?
Related page(s) Links to other wiki pages or feature requests.
Comments
Problem summary 7. Player does not understand what is the best weapon to attack, when having multiple options.
Problem description When new player has multiple weapons at his disposal, he may not choose the best one for the job. Sometimes he just picks the one he likes (swords look cool) or chooses it at random.
Screenshot(s)
Scenario(s) New player starts “Lesson 2: Monsters and Combat”, after picking up quarterstaff and killing few training dummies and a rat, he finds elven short sword. This sword is there because it is needed for chopping food, player wields it and starts to fight with it not understanding that the quarterstaff is better to fight with.
Breaks heuristic(s) 9. Does not provide adequate training and help.
Severity 3. At first we ranked this as 2, but the tutorial never explains how the weapon attributes works. This could potentially mean that even experienced players don’t fully understand them.
Occurred in Test one.
Suggestion(s) Add information tile to the tutorial explaining weapon attributes and descriptions.
Related page(s)
Comments
Problem summary 8. Player missing information tiles
Problem description New players who are used to “easier” games with self explanatory controls tend to play the tutorial in “rushmode”. They don’t realize how important some of the information tiles are and how detrimental missing the information can be. Although this problem cannot be fully repaired, missing the information tiles could be made harder.
Screenshot(s)
Scenario(s) New player plays the tutorial “Lesson 2: Monsters and Combat” and misses the information tile explaining how food is chopped. He may continue the tutorials towards higher lessons, but never learns how to chop food.
Breaks heuristic(s) 9. Does not provide adequate training and help.
Severity 4.
Occurred in Evaluation, pilot, test one, test two, test three, test four.
Suggestion(s) When going to information tile, player needs to press space to continue. Making a information tile explaining information tiles. The game should repeat vital information if it notices that the player doesn’t understand some feature (for example, when at low health, the player should be prompted to rest)
Related page(s)
Comments
Problem summary 9. The player doesn’t learn how to cast spells
Problem description In the spellcasting tutorial the player is given a dozen or so scrolls. He should read all these scrolls to learn how to cast spells - players don’t realise this, casting each scroll only once and then dismissing them as useless (or think that they may be useful later).
Screenshot(s)
Scenario(s) Occurs in the spellcasting tutorial
Breaks heuristic(s) 9. Does not provide adequate training and help.
Severity 4. Makes finishing the tutorial impossible
Occurred in All tests (excluding one?).
Suggestion(s) The player should start with some experience in spellcasting, requiring only a couple of scrolls to be read. He shouldn’t be able to proceed to the spellbook before he has skills to cast from it.
Related page(s)
Comments
Problem summary 10. The player fails to stop levitating
Problem description The player chooses to invoke the levitation ability using commands ‘a’ and ‘f’ and then chooses to stop the levitation. The key ‘f’ in the ability view has been replaced with key ‘g’ and the player has to manually search for the correct ability.
Screenshot(s)
Scenario(s)
Breaks heuristic(s) 6. Provide intuitive and customizable input mappings.
Severity 0. - 1.
Occurred in Test two.
Suggestion(s) Perhaps not an usability problem per se, but the most logical functionality would be toggle-like, ie. the same key would invoke and release the ability.
Related page(s)
Comments
Problem summary 11. The player doesn’t understand the context of different weapon and character statuses
Problem description There are a multitude of weapon and character statuses in the game, such as cursed, hungry, full, encumbered. The player fails to understand what these mean. The results range from humorous to frustrating.
Screenshot(s)
Scenario(s) Can occur in any of the tutorials.
Breaks heuristic(s) 9. Does not provide adequate training and help.
8. Provide users with information on game status.
Severity 2.
Occurred in Test one, three.
Suggestion(s) There should be a quick tutorial where the different statuses are explained.
Related page(s)
Comments Cursed status should be explained more clearly when you try to unequip a cursed item.
Problem summary 12. The player doesn’t understand the GUI
Problem description This is not a very tangible problem, but can clearly be seen when analysing the tests. Actually the problem is very multi-layered. You could say that the GUI has four major elements: Character status, message log, inventory and the actual play area. New players don’t understand the information given to them in the character status area, and neither do they understand the extreme importance of the message log.
Screenshot(s)
Scenario(s)
Breaks heuristic(s) 9. Does not provide adequate training and help.
Severity 2. - 3.
Occurred in All tests.
Suggestion(s) The tutorial should be in 6 parts, with a quick tutorial added to the 1 slot. This quick tutorial should only be a slide show of pictures, introducing the basic concepts of roguelikes, DCSS and the GUI.
Related page(s)
Comments
Problem summary 13. The status texts are confusing
Problem description Status texts like player health, hunger and inventory overweight are shown in same row in GUI. For example status text “Full” means that player isn’t hungry. Player thought that the inventory is overloaded and the levitation power cannot be used.
Screenshot(s)
Scenario(s) Occurs in the items and inventory tutorial
Breaks heuristic(s) 8. Provide users with information on game status.
9. Provide instructions, training, and help.
Severity 1.
Occurred in Test one.
Suggestion(s) Different type of status texts (hunger, inventory weight, health, etc.) should be separated.
Related page(s)
Comments
Problem summary 14. The Yes/No prompts are somewhat inconsistent
Problem description Some of the prompts accept both lower- and uppercase ‘Y’ and ‘N’ while some only accept uppercase letters.
Screenshot(s)
Scenario(s) The prompt “Do you want to leave the dungeon?” only accepts uppercase letters while “Really attack with while wielding your bread ration?” accepts both.
Breaks heuristic(s) 1. Provide consistent responses to the user’s actions.
Severity 1.
Occurred in Evaluation, test four.
Suggestion(s) Major decisions (at points where action of the player could cause something irreversible) could be prompted with “yes” while others would accept both lower and uppercase letters.
Related page(s)
Comments
Problem summary 15. The player dies and is forced to replay the tutorial
Problem description Death sometimes comes easily, resulting in the player replaying the tutorial. This causes frustration and ultimately ends in player quitting the tutorial.
Screenshot(s)
Scenario(s) New player plays the tutorial too aggressively
Breaks heuristic(s) 5. Allow users to skip non-playable and frequently repeated content.
Severity 3.
Occurred in Test 1 (resulted in quitting), but frustration could be found in all tests.
Suggestion(s) Make dying much harder in the early tutorials. Death should only be possible (probable) in the last rooms of each tutorial. It should be made clear that these are indeed the last rooms. Include checkpoints or chop the tutorials into even smaller pieces to avoid players replaying the same content again and again.
Related page(s)
Comments
Problem summary 16. Pointless, repetitive rooms in the tutorial
Problem description There are some rooms in the tutorial, which serve no apparent learning purpose. Killing easy enemies (such as rats) doesn’t really teach the player anything once he’s learned basic combat skills in part 2. This makes the tutorial longer for no apparent gain. In our tests, none of the test persons were able to beat the tutorial in the appointed 1,5 hours.
Screenshot(s)
Scenario(s)
Breaks heuristic(s) 5. Allow users to skip non-playable and frequently repeated content.
Severity 2.
Occurred in Pilot, all tests, expert reviews.
Suggestion(s) Remove pointless repetition. Just give the player rations instead of forcing him to kill rats.
Related page(s)
Comments

Post-test interviews concluded

As expected, couple of the interviews pointed out the difficulty of remembering all the relevant controls in the game; this manifestated itself in various forms: for example after the first test the player said that he felt himself a awkward player as he couldn’t use controls fluently. Having several controls for the same action also made the tester frustrated, he theoretized that if the game had been more restrictive with the controls, he had learned it better. This, however, can’t be generalized as other tester enjoyed the option to freely choose from the available controls.

Also one of the testers got confused with the double commands like ff, cases like these would have been improved if the previous input had been shown to the player somewhere in the screen. Customizable controls was suggested, however this isn’t easy to implement in a roguelike - at least in its full potential as roguelikes are usually featured with huge amount of different commands.

Dying within the game seemed to be another polarizing aspect; some of them felt it was natural (“- - death is a straightforward method to teach the game mechanics”) and some were frustrated by it. One small yet interesting point of improvement was a suggestion to include progress bar within the tutorials. This could have increasing factor for the awareness of the player as, for example, he would realize the looming of the final battle behind next door. The progress bar might also encourage the player to try a certain part again more actively; currently dying repeatedly on the same spot and not knowing the progress might lead to simply ignoring the part.

In conclusion, the reception was good. Most of the players admitted that they could try the actual game, a few, however, within the condition that playing would include a social aspect, eg. playing with a group or with a friend. Despite the slightly primitive tile graphics of the game, one of the players did seem to be really immersed within the game, when the moderator asked him about the best part, he straightforwardly (and humorously, of course) answered “The best part was running in a berserk rage and tearing the enemies into little pieces, standing on their corpses and listening to the weeping of their wives”.

What did we learn?

One of the major flaws was the time limit which was assigned to complete the tests. The limit itself ranged from 1 to 1,5 hours: however none of the testers could complete the tutorial within this limit. Hence this was enough to prove the existing gap between an experienced player and a novice: the former completes the tutorial within approximately 20 minutes while the latter may need as long as two hours to finish. However, one hour seemed sufficient when the tests were planned, bearing in mind that the expert finish-time was used as reference point. Considering now, the time limit should have been set to approximately 2 hours.

Other point of frustration was the slightly malfunctioning test equipment; the audio was slightly off and corrupted in some of the screen capture videos making the analysis difficult at certain parts. Also one of the test videos was severely lagging because two instances of the screen capturing software were accidentally run. While the latter problem is solely the fault of one team member, the former should be fixed with more robust choice of software. Other point of improvement might be the option to multiplex the screen capture and the video camera output into single video; this would ensure the fluency of the analysis.

The learning curve in roguelikes is quite steep as even the most basic concepts may be new to player. For example, one of the testers wasn’t familiar with the turn-based gameplay and realized it after some time had passed: “Well… If I just stay here, do I lose my turn?”. Also the “- - More –” dialog in the bottom of the screen was unnoticed when the tester got the impression that the game had crashed as nothing was happening: “- - The game seems to crash at this point; I have killed the rat but it just doesn’t disappear.. Clearly a bug in my opinion”. One of the most humorous remarks was when the player wielded a cursed dagger and thought it as a favourable item “I thought that the weapon had some ‘cursed’ enchantment in it and got excited when I noticed I had the item - - “.

One interesting concept would have been to test the ASCII graphics version of the game, which doesn’t feature the fancier graphics and mouse input. This would have ensured that the player is limited to strict set of controls, the keyboard. However, this approach woudn’t have been an easy way out as for novice the console version may feel even more intimidating than the tile version with its minimalistic outlook.

The tests were interesting and provided an insight to different playing styles and thought patterns of the testers which varied from trial-and-error approach to analytic and careful. However the most valuable single fact derived from tests might be the difficulty of designing a satisfying and challenging game tutorial; an aspect one player considers worthless and frustrating might be fun and rewarding for another. While this document is unable to emphasize any particular design approaches as such, it hopefully sheds some light into the most problematic areas of the tutorial.

Appendices in Finnish (Liite 1-3)

(Not converted to DokuWiki format yet. Please see below for the full report.)

Original PDF Report

Below is the full Dungeon Olms usability testing report as a pdf file.

The Dungeon Olms report (pdf)

Logged in as: Anonymous (VIEWER)
dcss/usability_project/dungeon_olms_report.txt · Last modified: 2011-11-30 15:18 by evktalo
 
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki