Designing the User Experience at Autodesk

Method 26 of 100: The Critical Incident Technique (CIT)

by Chauncey Wilson

The critical incident technique (CIT) was first described by Flanagan (1954), as a method for collecting negative and positive incidents that contributed to the success or failure of a task.

Critical incidents are brief and memorable descriptions of actions that a person or group performs in particular situations that lead to either effective outcomes (successes) or ineffective outcomes (failures or near misses). 

Ideally, critical incidents are real incidents reported by actual users doing real work in their normal work environments.

The word “critical” often refers to a severe usability problem or a clear success. But sometimes a critical incident is something relatively minor, like an error message on a new system that forces you to waste time debugging when your other tasks are more important.

The CIT is useful for improving software documentation, software applications, web sites, aircraft controls, surgery, product requirements, and business practices.

  Information to include in a critical incident report

• A description of the task in which the critical incident occurs. 
• A description of the critical incident.
• A statement indicating whether the action that is performed during the critical incident is effective (a success) or ineffective (a failure).
• Optionally, a rating of how critical the incident is for completing a task.

 Reasons to use the CIT

• Understand less frequent events that are not always evident in usability testing or other forms of evaluation where a person is observed for a relatively short period of time.

• Collect critical incidents from remote users with early working prototypes, current products, or competitive products when you cannot view the users directly.

 Strengths and weaknesses:

+  The CIT is relatively inexpensive and provides rich qualitative data.

+  Data from a critical incident study can reveal system weak points, and system strengths.

+  The flexible method is adaptable for interviews, observation, surveys, think-aloud testing, online discussion groups, and other methods.

+  The CIT sometimes identifies rare events that are not picked up by other methods that focus on common or everyday events.

+  The CIT is applicable across a wide variety of products, environments, and users.

-   The CIT method focuses on critical incidents, and doesn’t capture routine task information that is often extremely important in system design.

-   The criteria that makes an incident “critical” is sometimes ambiguous. 

-    People are sometimes afraid to report critical incidents where their behavior contributed to or caused a serious problem.

 Procedure for using the CIT

1. Define the system of interest and the length of your study. If critical incidents are rare, consider running your study for days or week.
2. Determine the method for collecting critical incidents.  Use paper forms, online survey tools, or online forms that are suitable for display when an incident occurs.
3. Create the critical incident form.
4. Distribute the form with brief instructions to the target participants.
5. Run the study for the period of time that is necessary to capture of rare events.
6. Compile the data and categorize incidents into common themes.
7. Organize related incidents by frequency.
8. Brainstorm solutions to the most frequent or most severe incidents.

References:

 Flanagan, J.C. 1954. “”The critical incident technique.”” Psychological bulletin, 51(4), 327-358.

The classic article is at:  http://www.apa.org/pubs/databases/psycinfo/cit-article.pdf

 UXPA. Critical Incident Technique (CIT). http://www.usabilitybok.org/critical-incident-technique

Method 25 of 100: The Monetary Method

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

The monetary method is an individual or group technique for prioritizing requirements by asking users and stakeholders to “buy” requirements from a “wish list” given a fixed (imaginary) budget of say “$1200”. Each requirement is described briefly, and a relative cost is associated with developing that feature.

This method for prioritizing requirements forces potential users to think carefully about what requirements are most critical. You can do this online without play money by just sending out the list and asking people to choose up to $1200 of items by checking cells on a questionnaire (and setting things up so no one can overspend).

Conte (2004) uses a variation on the monetary method as part of a rapid task analysis. Conte had a group of users write down, in 10 minutes, all the tasks that define their work on index cards - one task per card. The users then grouped the cards into different categories of work and listed task frequency (low, medium, high, really high) and task importance for each item (low, medium, high). Then the participants were given $1000 in fake $100 bills and everyone was asked to pin one or more of the $100 bills on the tasks they wanted the new product to support.

When to Use:
You can use the monetary method to:

• Prioritize requirements
• Learn what general sets of requirements, features, content, or other items provide the most value to different groups
• Get users to make clearer distinctions about what requirements are essential
• Make tough trade-offs when you need to drop features  

Strengths and Weaknesses:

+ Low cost, and minimal training

+ Fun and engaging

+ Can be conducted in person or online

+ The process of asking people to buy a set of requirements on a fixed budget provides a clear differentiation between important and “nice-to-have” requirements

+ Data analysis is simple

– You need a large sample of potential users

– Some companies worry about doing this online since it could reveal something about future products to competitors

– This method gets more complex if you have to consider dependent requirements

Procedure:

The basic monetary method procedure is:

1. Describe a small set of core requirements that will definitely go into a new version of software.  These “must have” requirements are guaranteed and not included in the list of requirements that users can buy. 
2. List a set of proposed requirements with clear descriptions and the relative costs to develop those features. For example, features that were easy to develop (e.g. 1-2 weeks) would cost $100, features that required a moderate development effort (e.g. 3-4 weeks) would cost $200, and features that were complex and difficult to develop (e.g. more than 4 weeks) would cost $300.  The figure below is a monetary method form that contains core requirements, instructions, and example requirements.
3. Brief the participants about the meaning of the relative costs.  For example, you might say that $100 requirements take 1-2 person-weeks to develop, $200 requirements take 2-4 person-weeks, and $300 requirements take more than 4 person-weeks of development effort.  
4. Give the participants play money and tell them that they could “buy” any combination of features as long as they don’t spend more than the allotted funds.  
5. Have users buy a requirement by circling (or choosing if online) the value in the Relative Cost column and put the money for that requirement aside. The amount of play money will depend on the number of features you present to the participants.  If you gave each participant $1200 in play funds, this means that they could buy four expensive features at $300 each, six medium priced $200 features, or various combinations of low, medium, and expensive features – as long as they don’t go over $1200. I generally give people enough money to buy three to five of the major features, though you can vary the amount of money to restrict the number of choices.  Another rough rule for assigning funds for buying requirements is that participants can’t buy more than about 30% or so of the total number of requirements with their funds.
6. Tabulate the results.

References:

Conte, L. (2004). The color of money – an agile technique for the prioritization of requirements. Proposal to the Boston UPA Miniconference. Natick, MA.

Gray, D., Brown, S., & Macanufo, J. (2010). Gamestorming: A playbook for innovators, rulebreakers, and changemakers.  There is a method in this book called the “$100 Test” that is similar to the monetary method.

Method 23 of 100: RITE Method

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 23rd in a series of 100 short articles about UX design and evaluation methods. Today, I will discuss the RITE method for finding and fixing usability problems.   The RITE method is a variation on traditional think-aloud usability testing where problems are fixed during the usability test phase and then retested to validate that the solutions improve usability. 

The RITE method, because of the rapid find and fix approach, is potentially a better match for agile development than traditional usability testing.

Method 23 of 100: RITE Method

The RITE method is generally attributed to Microsoft® Games Studios and Microsoft Research. The RITE method uses representative users, real tasks, observation of participant interaction with the product, and thinking aloud. The primary differences between traditional usability testing and the RITE method (Medlock, Wixon, Terrano, Romero, & Fulton, 2002; Medlock, Wixon, McGee, & Welsh, 2005) are:

• Fixes are made as soon as a problem and clear solution to that problem are identified.  An obvious problem with a clear solution might be fixed after a single participant.
• The product team must be able to fix problems quickly – say within a day.
• The team must agree on what tasks every user of the product must be able to perform to help set the priorities for problems that emerge during the testing.
• The product team must include time at the end of each testing day to make sure that fixes to problems have improved the usability of the system and have not introduced any new problems.

There is no fixed number of participants. Iterations often consist of 1 to 3 participants.  The testing goes on until all fixes are verified to improve usability.

When to Use:

You can use the RITE method to:

• Find and fix problems quickly
• Support an agile process
• Involve the entire team in making the product usable

Strengths and Weaknesses:

+  Gets entire teams engaged in the find and fix cycle

+  Keeps decision-makers directly in the loop about the usability of the product

+  Provides rapid identification or problems and solutions 

-   Strong recruiting support is needed to keep a constant flow of participants

-  Requires a strong team commitment to have the entire team go through the RITE process

-  The software/UI architecture of the product or service may not easily allow quick changes

-  Rare problems may not emerge during RITE testing

-  If there are many problems and many fixes, some solutions might break the same or different parts of the product

Procedure: 

The basic RITE procedure is:

• Get commitment from the product team and decision makers to observe test sessions.
• Get the team to agree to fix usability problems as they emerge from testing.  Include time in the testing schedule to fix emergent problems.
• Define the metrics of success.
• Choose representative users, define key tasks, and develop the testing protocol.
• Conduct the test sessions using a think-aloud approach.
• Analyze the data after each session.
• Review the problems that emerge after a session.
• Brainstorm solutions to problems.
• Agree on the best solutions and fix problems. If the team is not sure that they understand the problem continue testing the current UI until the problem is clear.
• Retest fixes and validate that the fixes did not introduce any new usability problems.

References:

Medlock, M.C., Wixon, D., McGee, M., & Welsh, D. (2005). The rapid iterative test and evaluation method: Better products in less Time. In Bias R., and Mayhew, D.J. (Eds.), Cost-justifying usability: An update for the Internet age (489-517). San Francisco, CA, USA: Morgan Kaufmann.

Medlock, M.C., Wixon, D., Terrano, M., Romero, R., & Fulton, B. (2002).Using the RITE method to improve products: A definition and a case study. Presented at the Usability Professionals’ Association, Orlando Florida.

Patton, J. (2008). Getting software RITE.  Retrieved July 15, 2013 from http://www.agileproductdesign.com/writing/ieee/patton_getting_software_rite.pdf

Shirey, J., Charng, A., & Nguyen, Q. (2013).  The RITE way to prototype.  UX Magazine. Retrieved July 15, 2013 from http://uxmag.com/articles/the-rite-way-to-prototype

Sullivan, B. K. (2011).  5 ways to improve your next agile project with the RITE method. Retrieved July 15, 2013 from http://bigdesignevents.com/2011/08/5-ways-to-improve-your-next-agile-project-with-the-rite-method/

Method 22 of 100: Affinity Diagramming

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 22nd in a series of 100 short articles about UX design and evaluation methods. Today, I will discuss affinity diagramming, a method for organizing qualitative data into related groups as an individual or group. This method can be used to make sense of data from lab studies, field studies, open-ended questions, brainstorming, diary studies and other methods that produce qualitative data. Affinity diagramming is often done by a group of people in a room with sticky notes, but you can also do this remotely with tools like Google® Spreadsheet (see below), card sorting software and Microsoft® Sticky Sorter.  

View the full article

Method 21 of 100: Collages

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 21st in a series of 100 short articles about UX design and evaluation methods. Today, I will discuss the use of collages for assessing emotional aspects of products and services and generating ideas and requirements.

Method 21 of 100: Collages
A collage study asks users to find pictures, words, and tangible objects from magazines, clip art, newspapers, the Internet, and physical artifacts that represent their feelings, emotions, and personal experiences with a particular product, service, setting, person, organization, or environment. The user is then asked to organize the clips into a physical or online collage. Tools for creating an online collage can be as simple as Microsoft® PowerPoint® or digital scrapbook software like MemoryMixer, My Memories Suite, and Digital Scrapbook Artist.

After creating the collage, the user, audience (in participatory design), and facilitator engage in a discussion about the meaning of the collage. Here are some examples of the use of collages in UX design:

• Asking people to construct a collage that illustrates their identity and aspirations:
• http://experientia.com/perspectives/creating-togetherbuilding-value-with-participatory-design/
• Making emotional connections:
• http://www.boxesandarrows.com/view/making_emotional_connections_through_participatory_design
• Using collages to share insights:
  http://studiolab.io.tudelft.nl/manila/gems/sleeswijkvisser/participatorycommunication.pdf

 While collages are generally focused on emotional aspects of product design, they can also be used to support early concept design or requirements gathering. For example, you might ask colleagues to create collages of different ways to use ratings in Web and mobile products and then to highlight the top three ideas. Collages can also be used to support future workshops where you might want to create an artifact showing how participants might expect their work to be different in ten years.

Part of the theory behind the collage method is that it draws out emotional and cognitive issues that direct interviews, observations, or surveys do not easily tap. Collages are probably best suited for research into products, services, or systems with which the participant has some direct experience or interest.  

Collage creation supports creativity, can be done by almost anyone, and requires little in the way of supplies or facilitation. Collages can be done by individuals or groups. Some UX practitioners use collages as a focus group activity.

One of the factors in the creation of collages is the degree to which the images are constrained. In some studies people have been given a set of pictures and/or words; in other studies, people had stacks of magazines or large online collections from which to draw their own images. McKay, Cunningham, and Thomson (2006) note that here is a trade-off between the number of images and the time required to create collages - a practical trade-off if you have limited time for creating collages.

Collages can be analyzed for themes. Themes might include things like:

• Privacy
• Security
• Usability
• Aesthetics
• Locations
• Relationships
• Barriers and support mechanisms
• Feelings about the target object or service

You can count the number of themes and also capture the collage makers’ comments and detailed issues, requirements, and problems.

When to Use:
You can use collage studies to:

• Understand emotional issues with concepts. For example, you might ask people to create a collage that expresses what is satisfying and dissatisfying about their work in general.
• Understand organizational culture. You might ask people to create a collage that describes aspects of their work culture and see images like barriers and army tanks or small images of smiling people in a conference room.
• Support participatory design (co-creation) workshops.
• Tie features and product attributes to user/customer values.

References:

Hanington, B., & Martin, B. (2012). Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Creative Publishing International.

McKay, D., Cunningham, S. J., & Thomson, K. (2006). Exploring the user experience through collage. In Proceedings of the 7th ACM SIGCHI New Zealand chapter's international conference on Computer-human interaction: design centered HCI (CHINZ '06). ACM, New York, NY, USA, 109-115.

Sanders, Elizabeth B.N., & William, C. T. (2001). “Harnessing People’s Creativity: Ideation and Expression through Visual Communication” Focus Groups: Supporting Effective Product Development. London: Taylor and Francis.

Method 20 of 100: The Magic Thing

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 20th in a series of 100 short articles about UX design and evaluation methods. This entry focuses on the use of a prop called the “magic thing” that can serve as an ideation tool in the design of mobile, ambient, and ubiquitous systems.  

Method 20 of 100: The Magic Thing
In a 2000 article, Guilio and colleagues describe various role-playing techniques that support the ideation and concept development of mobile services and devices. In their research, they asked participants to carry a “magic thing” as they walked around different locales and use it “to imagine how a portable device could support [them] in a particular situation” (p. 199). The rationale was that the use of a prop might provoke some novel ideas that would not emerge sitting around a table doing traditional group brainstorming.  One common approach for this method would be to give people scenarios and then let them walk around a particular area like a city or building to imagine what kind of “magic” a portable device could provide.  One scenario might be to find a restaurant in a city that you have never visited before that serves a certain kind of food and is quiet.  Another simple scenario might be to envision software that would help you, a brand new employee, locate people with specific kinds of knowledge in a large office.  In the graphic below a magic thing is used to evoke ideas about how context-aware devices might help someone in a strange city.

According to different sources, Jeff Hawkins, one of the inventors of the PalmPilot PDA, carried a small block of wood about the size of the Palm device with him to help envision how the device might benefit him at work (e.g. to generate a to-do list or use a calendar) and at home (e.g. to create a shopping list). The block of wood was his “magic thing” and provided inspiration for a generation of mobile devices. 

Howard and his colleagues (2002)  described “endowed props” that are rough physical simulations of devices that can used in role-playing  The props can be used to provide some constraints on the ideas participants generate so the results are “plausible science” rather than science fiction. Howard et al. suggest endowing props with particular capabilities like speech recognition, one at a time, so as not to provide too much magic to the participants.

When to Use: 
You can use a magic thing to:

• Serve as a rough example of the form factor of proposed devices (e.g. a pen, tablet, phone, or clip-on device)
• Complement scenario creation exercises
• Add some fun and minimal constraints to role-playing studies
• Provide a triggering function for ideas

Procedure:
The two articles referenced below provide several variations on role-playing or scenario enactment that involve magic things or endowed props. Here is a simplified procedure adapted from Howard, et al.

1. The design/research team introduces the participants (the actors) to a scenario that they are to act out.  For example, the scenario might have you lost in a city like Rome after missing your tour bus.
2. The participants review the scenario with a member of the design team and personalize it to their own situation.
3. The participants act out a scenario without the magic thing.
4. The participants are given or choose a prop (their magic things).
5. They are told what magical powers it has (to start). For example, the magic thing knows where it is (e.g., has GPS) and has information about commercial establishments and ratings of service for many companies.
6. The participants and design team then engage in participatory role-playing where the magic thing can acquire new powers and lose old ones.  For example, you might start with a magic thing that is location aware with information about services and then add a magic power like augmented reality where you visualize information as an overlay over camera images.
7. The sessions are recorded and the ideas that emerge are captured and applied to a new round of role-playing that expands on promising ideas.  Here you take good ideas about how to build magic into a portable device and do a deeper level of role playing to get more specific ideas.

References:

Giulio Iacucci, Kari Kuutti, and Mervi Ranta. 2000. On the move with a magic thing: role playing in concept design of mobile services and devices. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (DIS '00), Daniel Boyarski and Wendy A. Kellogg (Eds.). ACM, New York, NY, USA, 193-202. DOI=10.1145/347642.347715.

Steve Howard, Jennie Carroll, John Murphy, and Jane Peck. 2002. Using 'endowed props' in scenario-based design. In Proceedings of the second Nordic conference on Human-computer interaction (NordiCHI '02). ACM, New York, NY, USA, 1-10. DOI=10.1145/572020.572022.

Method 19 of 100: Laddering Questions

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 19th in a series of 100 short articles about UX design and evaluation methods. Today, the focus will be on the laddering question method, a semi-structured interview technique where you ask “why, how, what, and ‘tell me more’” questions multiple times to drill down, up and sideways to go from general to specific, specific to general, or from one item in a category to another item.

Method 19 of 100

Laddering questions can provide insight into attitudes, beliefs, and values that affect requirements, brand image, consumer preferences, and buying decisions. For example, you might be interviewing someone as she works with software in her office. During the interview, the person says, “This feature is great!” Then you would follow up with a “ladder down” question – “Why is that feature great?” The person might say “It saves me a lot of time.” Then you continue with another ladder-down question – “Why does it save you a lot of time?” The person replies that “I didn’t have to find the dialog box – I could just use the pop-up menu.” You could go further, but in this series of “why” questions, you have gone from a general emotional statement to one that gets at the underlying issue – a particular feature reduces the amount of movement and search time required to perform a task. Laddering is a way to move from surface statements to root causes, details that are actionable, or core values that might affect adoption or satisfaction or loyalty.

In addition to drilling down, you can also drill sideways and ask a set of questions that reveal multiple examples or categories related to a particular topic. For example, if you were interviewing a usability expert about what methods he uses to discover usability problems, he might say “usability testing.” You could then ask “Are there any other methods that you use to get at usability problems?” He would pause and say “pluralistic reviews.” You could drill down on “pluralistic reviews” and then drill sideways and ask for other methods that involve users, getting the answer “diary studies.” A combination of drill-down and drill-sideways questions can provide useful and persuasive data.

There are a number of methods that are based on laddering questions including the “5-Whys” method that is often associated with root cause analysis.   

When to Use:

You can use laddering questions to:

• Understand organizational culture
• Complement other methods
• Elicit and understand knowledge from experts
• Elicit knowledge and details about a new domain
• Determine details about how new features and product attributes affect the user experience
• Produce data that are actionable
• Link features and product attributes with user/customer values

Strengths and Weaknesses:

+ Laddering is a technique that gets at core values and the underlying reasons for particular behaviors or choices.

+ Laddering is a way to elicit semi-tacit information that might not be revealed by other UX methods.

- Laddering can be tiring for participants who are continually asked “why” or other laddering questions.

- Conducting interviews with many laddering questions requires a facilitator who can keep the participant engaged. People who use laddering questions need to be comfortable with the repetitive questioning style and be prepared to vary their prompts as needed.

- On some occasions, laddering can get into sensitive issues (e.g., how corporate politics affects productivity).

- Laddering questions assume that there is hierarchical information. In some domains, the information may not be represented hierarchically.

Procedure:

You can conduct interviews that are focused on laddering or you can use laddering questions as part of a more general interview protocol. Here is a high-level set of steps for using laddering questions.

1. Develop a small set of laddering probes that you can use in your study. Consider problems for moving down, up, and sideways. Here are some ideas for laddering questions:
   
   “Why do you think that this product is “great”?  (laddering down when a person makes a general statement and you want to get an underlying value, attitude, principle or rationale)
   
   “Why is this important to you?” (laddering up when a person mentions a specific feature and you want to understand the higher level value of the feature)
   
   “Can you tell me about similar/different types of….?” (laddering sideways to get examples or variations)
   
   
2. During an interview, you can elicit features, attributes, concepts, or constructs of interest. You could, for example, ask a person to list the features that she disliked the most or that are most important to productivity.
3. Ask the person to prioritize the list and then begin doing a “laddering interview” starting with the items at the top of the list. The person might tell you that she most dislikes the collaboration aspects of the product. You could then ask “Why do you dislike the collaboration aspects?”
4. Ask laddering questions until you get answers that have reached “the end of the ladder” or that are actionable.
5. Repeat the laddering exercise with the next item in the list or the next item in the interview where laddering is useful.

References:

Corbridge, C., Rugg, G., Major, N.P., Shadbolt, N.R. & Burton, A.M. (1994)
Laddering: Technique and Tool Use in Knowledge Acquisition.
Knowledge Acquisition, 6, pp. 315-341

Hawley, M. (2009). Laddering: A research interview technique for uncovering core values. UXmatters. Retrieved on January 20, 2012 from http://www.uxmatters.com/mt/archives/2009/07/laddering-a-research-interview-technique-for-uncovering-core-values.php

Rugg, G., Eva, M., Mahmood, A., Rehman, N., Andrews, S. & Davies, S. (2002)
Eliciting information about organisational culture via laddering. Information Systems Journal, 12, pp. 215-229

Sherlock, G. & Rugg, G. Using laddering and on-line self-report to elicit design rationale for software Proceedings of EASE & PPIG joint conference, Keele University, 8-10 April 2003 Addendum, pp. 453-472

The ‘5-Whys’ Method. http://www.mapwright.com.au/newsletter/fivewhys.pdf

---

What’s Next in the Series?

In the next post, I’ll describe the method of “Magic Things.”

Method 18 of 100: The User/Task Matrix

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 18th in a series of 100 short articles about UX design and evaluation methods. I’m back from my sabbatical and ready to keep building the UX methods toolbox. Today, we’ll look at a simple task analysis method called the User/Task Matrix.  

Method 18 of 100

Understanding the frequency and importance of tasks that are performed by different classes of users (or personas) is critical for many design decisions. For example, if an architect will be using a function hundreds of times a day for weeks on end, then you would want to design a highly efficient user interface that included shortcuts and ways to select many things at a time. In contrast, you might have some features, like creating templates, that you do occasionally and designing the UI for templates might require support for fast learning and “re-learning” after gaps of use. If you are designing something that you only do once a year and that task is critical, you might design a step-by-step procedure. A simple approach for exposing frequency and importance by user class is the user/task matrix. This approach can also indicate which tasks are primarily the responsibility of a single user group and which ones cross groups.

Here is a generic example of a user/task matrix. The letters represent task frequency: H = high; M= medium, and L = Low. Task 4 in the matrix is a critical task and is marked with a red background. Only one user group performs that critical task with low frequency so the interface related to that task should focus on preventing errors and be easy to learn.

User Groups	User Group 1	User Group 2	User Group 3	User Group 4	User Group 5	User Group 6
Task 1	H	H	M	M	M	M
Task 2	L		L	M	H	M
Task 3	L	L	L	L	M
Task 4		L
Task 5	L	L				L

When to Use: 

You can use a user/task matrix to:

• Summarize data related to frequency and importance of tasks.
• Determine whether some tasks are used by a few types of users or many users. In the example matrix, tasks 1, 2, and 3 are used by almost all groups with varying frequencies. This information can be helpful for making trade-offs about what features should get priority during the design and development cycle. For example, something that is important, done by many user groups, with moderate to high frequency, might be an area to focus usability sessions.
• Consider whether the primary focus for design should be on ease of learning or ease of use (efficiency).
• Determine if different classes of users can be combined into a single class.
• Provide some clues about where to focus user assistance development.  

Procedure:

1. Ask questions about task frequency and importance when conducting user research. 
2. Develop a set of user groups or personas that represent the major classes of users. Keep in mind that there are some low frequency tasks (like initial deployment of software across an organization) that are critical to success and often performed by a small group of users.
3. Develop a visual coding scheme for highlighting frequency and importance in your matrix. My example is very simple.You can extend the basic concept in my example to create something visually compelling. 
4. Use the matrix as input to feature and UI design trade-offs and other design activities. A related method called the UXI matrix was presented at the 2011 UPA conference by Jon Innes and James McElroy and discussed by John Schrag at http://dux.typepad.com/dux/2011/06/the-forgotten-user.html

 

Strengths and Weaknesses:

+ The user/task matrix is simple and conveys critical information for design teams who are considering what aspect of usability they should focus on during conceptual design

+ You can easily substitute the language of your organization into the matrix. For example, you might substitute “stories” into the task column and “personas” into the user group row.

-       If you have hundreds of tasks, the matrix can become unwieldy and you might have to create a hierarchical version.

References:

Hackos, J. T., & Redish, J. C. (1998). User and task analysis for interface design. New York, NY: Wiley.

Wixon, D. & Wilson, C. E.  The Usability Engineering Framework for Product Design and Evaluation. Handbook of Human-Computer Interaction (Second Edition). Elsevier: Amsterdam, The Netherlands, 1998, pp. 653-688.

 

---

What’s Next in the Series?

In the next post, I’ll describe an interview technique called “laddering” which is used to get at root causes, understand how knowledge is organized, and get insight into organizational cultures.

Method 17 of 100: The User Interface Race

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 17th in a series of 100 short articles about UX design and evaluation methods. Since I am about to go on a 6-week sabbatical, I thought that it might be fun to look at a method Ben Shneiderman called “the user interface race”. This is an entertaining method for examining how experts solve problems under substantial time pressure and public scrutiny.

Method 17 of 100:

In the old, but classic book, The Art of Human-Computer Interface Design (edited by Brenda Laurel, 1990), Ben Shneiderman wrote a short chapter on user interface races. In the user interface race, participants (generally experts whose egos are secure) compete against the clock and other criteria to accomplish a specific goal. The results of the race are then used to improve the quality of products and promote usability and good design.

When to Use:

You can use user interface races to:

• Reveal how users work under pressure.
• Evaluate competitive products.
• Promote good usability in a public and entertaining way.
• Provide insights into shortcuts and tasks strategy employed by power users (a group often neglected in traditional usability evaluations).

Procedure:

Planning for the Race

1. Choose participants who are open to public competition.
2. Choose a task for your race that is challenging for participants and engaging for viewers.
3. Develop the criteria for judging task performance. For most tasks, you might just use time, but you might also include a panel of judges to rate the quality of the solution (like the Olympic panels that judge divers and skaters).
4. Choose the judges for the competition and train them in the race method and ground rules.
5. Develop the tasks for the race. Keep the tasks challenging, but make sure they fit in a reasonable amount of time.
6. Develop the ground rules for local and remote viewers of the race.
7. If you are testing products that are new to the participants, give them equal amounts of time to “train for the race”.

Conducting the Race

1. Welcome viewers and participants to the race. Review the procedures and explain the task and scoring systems.
2. Have the participants read and sign consent and non-disclosure documents if they are from outside your company.
3. Provide time for the participants to warm up with the system if they are not using their own hardware and software.
4. Give the race participants the task and ask them to read and ask any questions. Avoid giving away any key tips.
5. When the participants are ready, give the signal to begin (Ready! Set! GO!)
6. When time is up or the participants have finished the task, Signal the end of the competition based on your rule for stopping.
7. Declare a winner and (optionally) award prizes (which could be gift certificates, plaques, or branded merchandise).

Strengths and Weaknesses:

+ Seeing people work under time pressure without much structure (e.g., the series of tasks in a usability test) can yield useful insights.

+ User interface races can be fun and be used to promote usability.

- There aren’t many cases in the literature about the user interface race method so some practice might be required before you stage a real event. Consider doing a trial run with friendly colleagues.

- I’ve run across people in workgroups who didn’t like the competitive aspect of the race method. You are likely to have a few colleagues who don’t like the concept of a race. You do have to choose participants who have relatively strong egos and who can accept winning or losing.

References:

Berkun, S. (2003). INTERACTIONARY - Sports for design training and team building. Retrieved on April 28 from http://www.uiweb.com/dsports/default.htm.

Keith S. Karn, Thomas J. Perry, Marc J. Krolczyk (1997). Testing for Power Usability: A CHI 97 Workshop. SIGCHI Bulletin Vol.29 No.4, October 1997. Retrieved on April 28, 2005 from http://bulletin2.sigchi.org/archive/1997.4/karn.html

Shneiderman, B. (1990). User interface races: Sporting competition for power users. In B. Laurel (Ed.). The Art of Human-Computer Interface Design. Reading, MA: Addison-Wesley. Pp. 221-224.

---

What’s Next in the Series?

I will be on sabbatical for the next 6 weeks. I may pen a post while I am away, but if not, I’ll continue with number 18 in October.

Method 16 of 100: Backcasting

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 16th in a series of 100 short articles about UX design and evaluation methods. Today’s method is “backcasting”. This is a method for futures planning that can be done in a workshop format.  

Method 16 of 100: Backcasting

Backcasting is a strategic planning method that asks a group of stakeholders to create ideal future scenarios and then work backwards as a group to figure out what is needed to get to the ideal states from the current state.

Frequently applied to sustainability, energy, and urban planning futures, backcasting has also been employed by UX design groups to model the user experience of a product 3, 5, 10, or even 20 years into the future.

When to Use:

You can use backcasting to:

• Develop visions of the future.
• Support strategic design projects that involve multiple products and users.
• Test assumptions for a complex, multi-year project that will certainly be the “next great thing”.
• Diverge from predictions that are conveniently extrapolated from what you already know (the forecasting approach) to freely envision a reality that is an extreme success.

 Procedure:

1. Find a location where you can stick things on a large wall or long table. You will need many small and large sticky notes in multiple colors.
2. Recruit a group of stakeholders and thought leaders. Backcasting is an activity that is easier if all participants are in the same room (though you could envision a future where this is done remotely and use backcasting to work out how to have a successful remote backcasting session).
3. Determine how far in the future you will start from; e.g., companies often develop 5-year plans that would be candidates for a backcasting workshop.
4. Establish a set of questions about the time frame, current state, future ideal states, actions, indicators, risks and opportunities that the facilitator will use during the backcasting exercise. Matthew Milan has a good list of questions on these topics starting on page 52 of his public slide show on backcasting (http://www.slideshare.net/mmilan/backcasting-101-final-public).
5.  Brief your stakeholders on the purpose of the backcasting session and the ground rules.
6. Conduct ideation workshops to identify the current state and future ideal states or scenarios.
   1. Describe the current state of the project, problem, or product. For example, you might explain the current user experience of a suite of products made by your company.
   2. Define one or more possible (and successful) future states. For example, you might envision futures with augmented reality, tablets, nano-technologies, ambient feedback, and brain-wave user interfaces.
7. Consider each future state and work backwards to identify actions, assumptions, risks, benefits, and other indicators that could lead to these future states.
8. Consult additional stakeholders who were not involved in the backcasting exercise and get their feedback.
9. Publish the results using charts, maps, stories, list of actions required, risk/benefit trade-offs, and photographs of the actual items produced during the backcasting exercise.

Strengths and Weaknesses:

+ Simplified backcasting can be done on the cheap in one or two-day workshops using sticky notes, pens, paper, and string.

+ Backcasting is a way to reach a common understanding of successful futures and the steps  required to achieve them.

+ Backcasting provides actionable information.

- Backcasting requires a strong facilitator and solid preparation.

-Valuable backcasting results can be easily lost, unless there is an ongoing campaign to keep the results in the eyes of the stakeholders.

References:

Backcasting. Retrieved on July 25, 2011 from http://wearearising.org/2009/01/13/backcasting/

Milan, M. Backcasting 101: Retrieved on July 20, 2011 from: http://www.slideshare.net/mmilan/backcasting-101-final-public

Quist, J., & Vergragt, P. (2006). Past and future of backcasting: The shift to stakeholder participation and a proposal for a methodological framework. Futures
Volume 38, Issue 9 , November 2006, 1027-1045

Wikipedia. Backcasting. Retrieved on July 20, 2011 from:http://en.wikipedia.org/wiki/Backcasting 

---

What’s Next in the Series?

The next UX method posting will be a summer surprise J.