Designing the User Experience at Autodesk

Method 15 of 100: Misuse Scenarios

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 15th in a series of 100 short articles about UX design and evaluation methods. Today’s method is the “misuse scenario”. This is a method that focuses on possible misuses, both unintentional and malicious, of a system.

 Method 15 of 100: Misuse Scenarios

Scenarios are used to focus product teams on user needs in specific contexts. There are a variety of scenario types that you might consider in design. Here are a few general categories:

• Typical scenarios - These scenarios focus on common tasks and contexts encountered by the major personas.
• Atypical scenarios - These are infrequent, but sometimes important, scenarios. An example here would be the scenario for the recovery of information after a major and extended power failure. Recovery after a disaster is rare and an atypical scenario might describe this rare event (sometimes called a “black swan”) and how it would be handled.
• Extreme (but not misuse) scenarios - Extreme scenarios often focus on scalability issues like what happens to a Web site when a particular event results in quadruple the number of people expected by the requirements document pounding your server.

I would add misuse scenarios to this list. Misuse scenarios might deal with events like vandalism or inadvertent damage.   An example where misuse scenarios might aid design would be public kiosks (e.g., mailing kiosks) that you find in museums or malls. What kinds of misuse scenarios might be relevant? Here are a few:

1. Vandals put superglue in the slots for credit cards, receipts, and stamps. (malicious misuse).
2. Parents put their infants on the scale designed to weigh packages of no more than 15 pounds (unintentional misuse).
3. A slightly intoxicated couple spills a drink during the science museum’s New Year’s gala (unintentional misuse).
4. People with the flu and other diseases handle the mouse and touch screen in a health clinic (inadvertent contamination of public computer systems).
5. An evil person gathers information from a web browser at a library that wasn’t cleared after someone left quickly to deal with a home emergency (malicious misuse).
6. The software is being used for something completely unintended (could be malicious or not).

When to Use

Misuse scenarios are probably something to consider on any project where there is potential for someone to perform either malicious actions or actions that result in an unintentional, but unfortunate damage to a person, organization or the product itself.

This method can be used early in the product design and development process as part of requirements gathering and conceptual design. Including misuse scenarios in the UX design process could lead to a system design that would minimize the damage caused to the system itself.

 Procedure

1. Consider typical or atypical scenarios or common use cases and convert them to misuse cases.
2. Tap field support or technical support colleagues and ask them to recount stories related to misuse of a similar product or system.
3. Consider aspects of the environment that might interact with the system and users to create a misuse scenario.   For example, consider how a new handheld device might be affected by a torrential rain storm or extreme heat or the actions of a person who found the device on the ground and was lacking in ethics.
4. Brainstorm with a range of colleagues about “how could things go wrong with our system?”
5. Create a matrix of misuse cases and brainstorm ways to eliminate or mitigate any threats.

Strengths and Weaknesses

+ There are often stories and bug reports that can be used to develop misuse scenarios. Using misuse scenarios strengthens the likelihood that the entire user experience is considered during design.

- It is sometimes hard to envision misuse scenarios for new products. Sometimes the focus is so much on typical scenarios, use cases or user stories, that there is little effort spent on formulating misuse scenarios and possible solutions.

- Misuse scenarios can be quite vivid and may prompt design teams to go overboard in designing against misuse.

References

Alexander, Ian. Misuse Cases: Use Cases with Hostile Intent. Retrieved on June 29, 2011 from http://perceval.gannon.edu/xu001/teaching/2010spring/GCIS504/termpaper/readingList/MisuseCasesHostileIntent.pdf

Alexander, Ian, Use/Misuse Case Analysis Elicits Non-Functional Requirements, Computing & Control Engineering Journal, Vol. 14, 1, pp. 40-45, February 2003

Wikipedia. Scenario. Retrieved on June 29, 2011 from http://en.wikipedia.org/wiki/Scenario_(computing)

---

What’s Next in the Series?

The next UX method posting will discuss backcasting, a technique for generating a vision of a desirable future and then working backwards to see what is needed to get to that desirable future.

Method 14 of 100: “Yes, and …”

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 14th in a series of 100 short articles about UX design and evaluation methods. Today’s method is “Yes, and …”. This is a technique borrowed from improvisational (improv) theater that can be used to generate ideas and requirements. The technique requires little training and can be used with small or larger groups.

Method 14 of 100: “Yes, and …”

In the world of improv theater, performers start with an idea or scene and then build on that by adding a bit of information. The process of adding new information is referred to as “Yes, and …” and this method allows performers to continually redefine the scene and make leaps into new dramatic (or comedic) territory.

The approach can be used in the UX domain as a way to explore possible user scenarios, generate requirements, brainstorm, or envision future concepts. To start the process, you provide an initial verbal scenario and then ask colleagues to say “Yes, and …” and then fill in some new information or add new requirements.

Here is a short hypothetical “Yes, and …” session undertaken by a group of design colleagues or foodies who are defining the concept of an electronic menu for a new restaurant:

• I am planning a new restaurant and want to use electronic menus that offer the diner all the information that a waiter or waitress could normally provide.
• (a colleagues says) “Yes, and the electronic menu must be water proof because kids might spill things on it…
• (another colleagues says) “Yes, and we need to provide pictures of our meals…
• Yes, and we need to a way to allow people to do substitutions…
• Yes, and we need to include a way for people to tell us about allergies...
• Yes, and the calorie count must be included – that’s the law now…
• Yes, and we need to know if this is a romantic dinner or that someone is going to propose…
• Yes, and when someone is going to propose, we can bring out our special cake and make sure this is something they remember. 
• ... and on and on and on with “Yes, and …”

When to Use:

This method can be used early in the product design and development process as a way to elicit requirements, explore concepts, or even help develop personas. This approach sometimes reveals gaps or missing requirements that might elude more structured approaches. It is also a fun and simple approach that can enliven a tired team.

 Procedure:

1. If you read the improv literature, you will find different approaches for the “Yes, and …” method, but the basic process is to assemble a group and provide a starting statement or scenario like the electronic menu example.
2. Describe the "Yes, and ..." method and provide a brief example.
3. Consider if there are props that you could use to trigger ideas, for example, in the electronic menu example, you might bring an iPad to the “Yes, and …” session.
4. Ask someone to write the ideas where they are visible to the group or tape the session so you capture all the details.
5. Have the leader of the group start the “Yes, and …” session with the introductory statement.
6. Encourage someone to say “Yes, and …” and add something new. If you need to, you can say “Yes, and …” to prompt the next reply.
7. Continue until the group slows down.
8. Thank everyone for their contributions and willingness to engage in the “Yes, and …” activity.

Strengths and Weaknesses:

+  The basic “Yes, and …” method as applied to UX does not require a lot of training (though if you are planning on doing improve for a living and will depend on humor, you will need a lot of practice to be perceived as spontaneous). As a side note, some training on improv and theater techniques could be quite useful for a UX group. There are many papers on the application of theatrical methods to product design.

-  I’ve seen this method get a bit out of control and you might need to intervene at some point and direct the group away from the ridiculous without seeming like a dictator. In improv theater, you would not generally engage in this type of blocking of the scene development, but in the product development world, sometimes you need to change a focus that has gone outside the bounds of usefulness or appropriateness.

References:

Danzico, L. (2010). From Davis to David: lessons from improvisation. interactions 17, 2 (March 2010), 20-23.

Gerber, E. (2009). Using improvisation to enhance the effectiveness of brainstorming. Proceedings of the 27th international conference on Human factors in computing systems (CHI '09), 97-104. 

Wikipedia. Improvisational theatre. Retrieved from http://en.wikipedia.org/wiki/Improvisational_theatre on June 1, 2011

 

---

What’s Next in the Series?

The next UX method posting will discuss misuse scenarios. These are scenarios that focus on threats, both intentional and unintentional, to a system.

 

 

Method 13 of 100: Consistency Inspection

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 13th in a series of 100 short articles about UX design and evaluation methods. Today’s method is “consistency inspection”.  This is a general method for examining one or more products for internal or external consistency. Every company I’ve worked for has had a goal to “make our products consistent,” but as I described in an earlier DUX blog, http://dux.typepad.com/dux/2009/03/the-consistency-conundrum.html, the concept of consistency is complex. The goal of this posting is to suggest a general method for finding and cataloging inconsistencies within a single product or between multiple products. Comments on this general method would be much appreciated.

Method 13 of 100: Consistency Inspection

When to Use:

Consistency inspections are useful when you get the “make things consistent” mandate from senior management, acquire new products that you will be integrating with existing products, or when your users are complaining about how consistency hampers learning and increases errors.

 Procedure:

1. Create a small team of “inspectors” with different backgrounds – design, usability, training, QA, user assistance, development, and product management. The rationale for a small team is that different perspectives will yield more consistency issues.
2. Define the target of the inspection. The target can be a particular area of a product, a product, or a set of products. Ask the inspectors to review specific areas of the target product against any consistency exemplar and references. In a major product with many features, you might assign sub-teams that will each look at a particular type of consistency.
3. Define an explicit list of what consistency attributes are the focus of the inspection. You may want to focus on a single attribute like icon/menu consistency or multiple attributes (e.g., terminology, layout, use of UI controls, workflow, work-context).
4. Provide the “exemplar” or reference against which you will compare within a product or between products. For example, you might have a style guide like the OSX or Windows 7 user experience guidelines with standards and guidelines that you use to determine where the product or products are inconsistent. You might designate a particular application as the exemplar – the product that you want all others to be consistent with – and note where related applications, like those in a suite, differ from the exemplar. You might also have a pattern library that you can use as a “consistency reference”.
5. Train your consistency inspectors on the following topics:
   1. The definition of consistency and the different levels (e.g., within a dialog, between dialogs, across one or more applications, with applications that are used with a particular application, with the Mac or PC operating systems) and attributes of consistency.
   2. Examples of “consistency bugs”. Here, you want to provide clear examples of items that you want to track as consistency bugs and have discussions to help the team grasp the range of consistency problems that will emerge.
   3. How to report consistency issues? A template for reporting might include: The product or products under scrutiny, the particular component (e.g., file open dialog or pop-up menu), the exemplar or reference that you using to determine if something is inconsistent, and the type of inconsistency.
6. Provide the inspectors with the target of the inspection, which can be a spec, a wireframe, a prototype, or working product. Early consistency inspections are highly recommended to avoid embarrassment late in development. For example, you might find that there are two different search engines in the same product that worked very differently – an embarrassing and costly consistency gaffe!
7. Categorize and prioritize the issues and work with the inspection team on a plan to improve consistency.

Strengths and Weaknesses:

+  This method provides some explicit guidance about what to look for and compare against when you are asked to review a product or set of products for the multifaceted concept of consistency.

+  Creating a team and training them on consistency can promote consistency across groups. For example, once developers or product managers do a consistency review, they will be sensitive to issues of consistency in the future.

-  Consistency is a complex characteristic since there can be many reference sources against which to make consistency judgments. In addition to this inspection method, you need to promote consistency in many ways (see the oldie, but good book, “Coordinating User Interfaces for Consistency” for many ideas on how to promote and improve consistency)

References:

De Oliveira, R. & Da Rocha, H. V. (2007). Consistency priorities for multi-device design. In Proceedings of the 11th IFIP TC 13 international conference on Human-computer interaction (INTERACT'07), Cécilia Baranauskas, Philippe Palanque, Julio Abascal, and Simone Diniz Junqueira Barbosa (Eds.). Springer-Verlag, Berlin, Heidelberg, 426-429.

Grudin, J. (1989). The case against user interface consistency. Commun. ACM 32, 10 (October 1989), 1164-1173.

Nielsen, J. (1989). Coordinating user interfaces for consistency. San Francisco: Morgan Kaufmann. [Note about the book at http://www.useit.com/jakob/constbook_preface_2nded.html]

Wilson, C. (2009). The Consistency Conundrum. http://dux.typepad.com/dux/2009/03/the-consistency-conundrum.html

---

What’s Next in the Series?

The next UX method posting will describe a method borrowed from improv called “Yes, and…” that you can use as an ideation technique for stories, scenarios, and requirements.

 

 

Method 12 of 100:Teachback

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

By Chauncey Wilson

This is the 12th in a series of 100 short articles about UX design and evaluation methods. Today’s method is called “teachback”. In the teachback method an expert explains concepts, workflow, procedures, or instructions, associated with a particular domain to a non-expert (who could be a user or an interaction designer). The non-expert then tries to teach back what the expert had explained. During the teachback session, the expert corrects any misunderstandings. The misunderstandings and the actual expert discussion are both data that interaction designers can use as input for product design.

As an example of how teachback can be used for design, consider an interaction designer with some basic knowledge of a product or domain. That designer is tasked to work on some new features that are complex, so she finds several experts and asks them, one at a time to explain work related to that new feature. She then teaches back what she learned from each expert and asks the expert to correct any errors or misconceptions. The interactions are recorded. After several teachback sessions, the interaction designer should have a much deeper understanding of the issues and concepts associated with the new features and probably discovered some use cases that had been missed.

Method 12 of 100: Teachback

When to Use:

Teachback is a technique for extracting information from experts that would be useful for gathering requirements, learning about workflow, and understanding mental models. The method is also used in healthcare to determine if patients understand instructions about medication or medical procedures. In the healthcare arena a medical provider, home health aid, nurse, or doctor, would describe a procedure to a patient and then ask them to “tell me in your own words how you would…”

Procedure:

The teachback requires a domain or subject matter expert, a non-expert (the “learner”), and, optionally, a facilitator who takes notes and handles the recording. The learner and the facilitator can be the same person.

1. A subject matter expert is asked to describe a concept, task, or something else about a particular domain or product. A facilitator records the sessions with audio or video (or just takes notes if electronic recording is not possible.
2. The non-expert (e.g., the interaction designer listens carefully to the expert.
3. The non-expert is then asked to “teach back to the expert” what the expert had previously described or explained.
4. When the non-expert is teaching back what she learned, the expert corrects any misinterpretations, errors, or simplifications.
5. The session is repeated with another expert who again teaches the non-expert.
6. The transcripts from all the sessions are examined for common misinterpretation, errors, or simplifications.

Strengths and Weaknesses:

+   The basic approach is simple. An expert discusses or explains something to a non-expert related to a particular domain or product. The non-expert then describes back what he/she learned.

+ The method can highlight concepts that are hard to understand, instructions that are difficult to grasp, or gaps in mental models.

- The teachback method is not highly structured. The expert can present information in different ways using different styles, which might create problems for the non-expert.

- The non-expert needs to have some basic awareness of the concepts for a product or process so the expert doesn’t feel that his or her time is being wasted.

- The non-expert might feel bad if the expert’s corrections are too harsh.

- If you choose an expert who is arrogant and obnoxious, you might have an unhappy learner and a wasted session.

References:

Johnson, L. & Johnson, N.E., (1987). Knowledge Elicitation Involving 

Teachback Interviewing in Kidd, A.L., (Ed.), 1987, Knowledge Acquisition for Expert Systems: A Practical Handbook.

---

What’s Next in the Series?

The next UX method posting will describe some ideas on how to do consistency reviews. This is a long-delayed continuation of an earlier column about consistency in software design. The next posting will be somewhat more general than earlier ones.

 

UXD Method 11 of 100: Bodystorming

by Chauncey Wilson

This is the 11th in a series of 100 short articles about UX design and evaluation methods. Today’s method is called bodystorming. Bodystorming is an immersive ideation method for exploring ideas through role-playing and physical interaction with props, prototypes, actual products, and physical spaces. The overall goal of bodystorming is to understand the relationships between people, their physical location, and the things (e.g., tools, devices, materials) they use in that environment.

If you were designing a postal kiosk for a mall, for example, you might go to a mall with a prototype system and role-play a scenario about trying to use the kiosk when you are holding a baby and trying to weigh a package. Another bodystorming scenario suggested by Oulasvirta, Kurvinen, & Kankainen (2003) involves role playing elderly users (with memory problems) as they try to remember product information in a supermarket or drug store.

Method 11 of 100: Bodystorming

When to Use:

Bodystorming is useful when you are designing devices or interior or exterior spaces. For example, you might use bodystorming to understand how users of different heights and ages would experience different versions of aircraft cabins (for example, what are the problems with lifting luggage in crowded planes from the floor to the overhead bins), or the layout of modern train cars. Bodystorming can be quite useful in understanding the experience of teams who work in close quarters like doctors and nurses in an operating room or the cooking staff in a restaurant. Bodystorming is a way to envision how people will interact with ubiquitous computing systems like smart homes and virtual meeting spaces.

Procedure:

There are different approaches to bodystorming. Here is a basic approach.

1. Get a small group together for the bodystorming.
2. Define the locations were a product or service will be used. Go to those locations and observe how people interact. Watch how people interact with each other and the artifacts in their environment.
3. Develop the prototypes and props that you need to explore an idea. You can do this on the cheap with cardboard, sketches, existing furniture, and whatever else you can find nearby.
4. Identify the personas or roles that are important for understanding your product, service, or environment. Keep in mind that people may play the role of hardware or software as well as the roles of customer, user, or troubleshooter.
5. Role play different scenarios. Feel free to improvise and role play new situations and scenarios that emerge from your initial round of bodystorming.
6. Reflect on the bodystorming experience. What did you learn? What new questions emerged? You are likely to find new possibilities as well as gnarly problems.

Strengths and Weaknesses:

+ The method is useful for generating empathy for users.

+ The method provides clues about the impact of the environment on the user experience.

-  Bodystorming involves role playing and improvisation which may be difficult for some members of a product team.

 

References:

Dennis Schleicher, D., Jones, P. & Kachur, O. (2010). Bodystorming as embodied designing. interactions 17, 6 , 47-51.

Fairbrother, A. (2008). Bodystorming. Retrieved on April 4, 2011 from http://annefairbrother.co.uk/2008/03/26/bodystorming/

Gray, D., Brown, S., & Macanufo, J. (2010). Gamestorming: A playbook for innovators, rulebreakers, and changemakers. O’Reilly.

Oulasvirta, A., Kurvinen, E., & Kanjaunen, T. (2003). Understanding contexts by being there: Case studies in bodystorming. Personal Ubiquitous Computing 7, 2, 125–134.

---

What’s Next in the Series?

The next UX method posting will describe a method called teachback that is used to elicit and understand information from subject matter experts. 

 

Method 10 of 100: Perspective-Based Inspection

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the 10th in a series of 100 short articles about UX design and evaluation methods. This means that this series of short postings on methods is now 10% complete and there are only 90 or so weeks until the toolkit project is complete. Today’s milestone method is the perspective-based inspection. A perspective-based inspection is a user interface evaluation method where the evaluators are asked to adopt a specific perspective as they examine a product for problems. For example, I might ask members of my UX team to review a product from the following perspectives:

• Consistency czar – this person looks for any consistency issues with the product.
• Disabled user – this person would role play a person with specific disabilities and note where the product is not accessible.
• Psychologist – this person would look for violations of psychological principles related to memory, learning, attention, fatigue, interruption, persuasion, etc.
• Super power user – this person would look at issues of efficiency, shortcuts, and aspects of the product to support the very expert and frequent user.
• Error expert – this person would focus on areas where users might make errors and places where the system could support error prevention.
• Artist – this person would look for problems with aesthetics like clutter, poor user of color, graphics that look amateurish, and ugly icons.
• The Keyboarder – this person does not use the mouse and evaluates a product for complete keyboard access.

While each person has a specific focus for the inspection, any person can report any issue.

Method 10 of 100: Perspective-Based Inspection

When to Use:

Perspective-based inspections are useful for broadening the problem-finding ability of both experienced and inexperienced evaluators. For example, if your team was composed of mostly young people, you might ask one or more team members to adopt the perspective of an elderly person. Adopting a perspective might involve reading about issues with older users, simulating the effects of aging by wearing glasses that blur vision, or spending time with some older users before the inspection to understand some of the issues they might face. Since I’m now 60 and officially old by some measures, I am aware of the issue with tiny text and that might be something a younger colleague would miss.

Personas can also be the source of different perspectives and a persona-based evaluation would be a variation on the perspective-based inspection. Perspective-based inspections of current or competitive products can be a source of requirements as well as problems. You might note in the requirements definition phase how well a product supports particular perspectives. This type of UX evaluation is good when you need a quick review, don’t have easy access to the necessary users, and are lacking travel budget. You can do this type of review at any time during the development cycle, from requirements to finished product. 

Procedure:

1. Define the perspectives that are important for the product or service you are developing. Some of the things to consider are:
   1. Existing personas. Each persona will have a different perspective.
   2. Frequency of use by users. Two perspectives that I’ve used are the “really naive user” and the “product guru”. One perspective gets at learnability while the other focuses on efficiency.
   3. International users. These users could be colleagues from other countries who would look at issues with icons, idioms, metaphors, layout, etc.
   4. Users with disabilities.
   5. Angry users of the current product.
2. Provide some background about the perspectives and consider ways to understand the perspective better. If you were simulating an elderly user, you might have the evaluator read several studies that describe user experience issues that confront elderly users. If there are particular heuristics or checklists for the perspectives, make those available to the team.
3. Define the tasks, goals, and specific characteristics that are important for each perspective.
4. Provide the inspection team with the forms to record the issues and problems they find.
5. Conduct a short training session to review the perspectives, procedures, and reporting forms.
6. Have the evaluators conduct their inspections of the product independently.
7. Pull all the issues together from the independent inspections and review the issues with the entire team to clarify the data and prioritize the findings.

Strengths and Weaknesses:

+ The method is relatively inexpensive.

+ This method can be conducted by evaluators at different sites.

+ The method broadens the types of problems that emerge.

+ The method can be accomplished using a variety of online tools.

- Some perspectives may be difficult to assume. For example, I once worked on a project where one of the perspectives was that of an elderly Nobel Prize winner and taking on that perspective required some effort.

- The evaluators have to be open to adopting different perspectives; some colleagues may have difficulty with role playing.

References:

A Perspective-based Technique for Usability Inspection. http://www.cs.umd.edu/projects/SoftEng/ESEG/topics/usability.html

Redish, G., Chisnell, D., & Lee, A. A new take on heuristic evaluation: Bringing personas, tasks, and heuristics together with a new model for understanding older adults as users. http://www.redish.net/content/talks.html.

Zhang, Z., Basili, V., and Shneiderman, B., Perspective-based usability inspection: An empirical validation of efficiency, Empirical Software Engineering 4, 1 (March 1999), 43-69.

---

What’s Next in the Series?

The next UX method posting will be a surprise. : )

 

 

Method 9 of 100: Reverse Card Sorting

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

by Chauncey Wilson

This is the ninth in a series of 100 short articles about UX design and evaluation methods. Today’s method is called reverse card sorting. Reverse card sorting (also called inverse card sorting, reverse lookup, or tree testing) is a variation on closed card sorting where you place cards representing content, tasks, or navigation items onto a predetermined hierarchy (or other type of structure) and then assess how often users placed the cards into the “right” categories on the hierarchy.

Method 9 of 100: Reverse Card Sorting

When to Use:

Reverse card sorting is a variation on closed card sorting where a user is presented with a hierarchical diagram and a pile of cards representing categories and subcategories of information. Participants are asked to place the cards at the correct level of the hierarchy. The primary metric for this method is the mean percentage of cards that are sorted into the correct location on the hierarchy. This method can be used to validate changes to Web site navigation, hierarchical menus, or other task structures. In one study by Human Factors International, reverse card sorting was used to compare the navigational structure of a new Web site design to the old structure. The result of the study was that “…96% of the users understand the new site’s organizations and task groupings, compared with only 45% on the old design.” 

Figure 1: Sorting cards from a pile onto a hierarchy

 

Procedure:

1. Create a structural diagram and enter the labels for the high-level categories (see Figure 1). For example, if you are testing the main navigation for a new Web site, you would create a diagram with the top-level navigation labels filled in.
2. Create a set of cards that have questions, tasks, or names of the items that go into the sub-categories of the structural diagram. For example, you might create a card with a question like “Your wife has just purchased an iPad. Where would you go to buy her the leather cover that activates the device?” You would then place that card under the appropriate category on the tree diagram. The cards should not use the same language as the actual category labels. For example, in the iPad example, the actual category might be “Accessory” so you would not want to use that term in the question that you have on the card.
   1. You can use a tool like PowerPoint to create the structural diagram and online cards and send participants the PowerPoint file with instructions for the reverse card sort.
   2. You could also use collaborative graphics tools like Gliffy or Google Docs to conduct remote reverse card sorting sessions.
   3. There is a tool called “Treejack” that examines the “findability” of items in a tree structure by asking a question and then asking users to navigate through a tree structure until they find the right page. For more information on Treejack see http://www.optimalworkshop.com/treejack.htm
3. Shuffle all the cards that will be placed on the diagram or arrange them in a random order if you are using an online sorting tool.
4. Ask the participants to place each card in the lower level category on the diagram where they would expect to find the information on the card or get the answer to a question. Tell the participant that there is no wrong answer and you are interested in understanding how people would group information.

Strengths and Weaknesses:

+ The method is simple; it is like a visual quiz.

+ The method allows an evaluation of an information structure independent of the visual design or other confounding factors.

+ The method is useful for evaluating the effectiveness of an information architecture.

+ The method can be accomplished using a variety of online tools.

-  Reverse card sorting can be difficult if the information structure is complex.

 

References:
Human Factors International (N.D.). Case study – corporate intranets. Retrieved from http://www.humanfactors.com/downloads/documents/arinc.pdf on March 14, 2011.
Paul, C. L. (2008). A modified approach to a new card sorting methodology. Journal of Usability Studies, Volume 4, Issue 1, pp. 7-30. Retrieved from http://www.upassoc.org/upa_publications/jus/2008november/paul1.html on March 14, 2011.
Thurow, S., & Musica, N., (2009). When search meets web usability. Berkeley, CA: New Riders.

---

What’s Next in the Series?
The next UX method posting will describe perspective-based user interface inspections – a method where reviewers evaluate a product from different perspectives to broaden their problem-finding ability.

Method 8 of 100 - Repeated Card Sorting

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

 by Chauncey Wilson

This is the eighth in a series of 100 short articles about UX design and evaluation methods. Today’s method is called repeated card sorting. Repeated card sorting is a simple method for uncovering important dimensions of a set of related items.

Method 8 of 100: Repeat Card Sorting

When to Use:

Repeated card sorting is a variation on open card sorting that involves the repeated sorting of a set of items to understand underlying dimensions or characteristics of a product or service (Curran, Rugg, & Corr, 2005). One use of the repeated card sort method is to see if the dimensions that usability specialists believe are most critical to product success match the dimensions that are important to actual users. In common usability surveys, we might use “standard” dimensions like satisfaction, efficiency, and learnability that may or may not be what is important to users. Repeated card sorting can provide some validation of those standard dimensions and suggest new dimensions that might be even more important those our standard ones. Upchurch, Rugg, & Kitchenbaum (2001) used a repeated card sort to determine what qualities of a Web site influenced its perceived quality. In their study, individual participants came up with 4 to 11 sort criteria each. Over 80 sort criteria, some quite different from basic usability dimensions emerged from the study. If you were to do a repeated card sort of online banking sites, you might come up with some of the dimensions below:

• Trustworthy – Untrustworthy
• Ugly – Attractive
• Simple – Complex
• Dense – Open
• Informal – Formal
• Small Customer – Large Customer
• Colorful – Lacking Color
• Many Ads – Few Ads
• Scary – Inviting

 

Procedure:

1. Provide participants with a set of cards (or pages) that are thoroughly shuffled.
2. Ask each participant to sort the cards into two piles.
3. Have the participant name each pile and describe how the piles are different. For example, the participant might name one pile “Modern” and other pile “Old Fashioned” and describe what those names mean.
4. Shuffle all the items again and ask the participant to sort the cards in a different way.  
5. Name the new groups and once again ask the participant to describe how the piles are different.
6. Continue for 5 to 6 rounds or until the participant cannot come up with a new set of piles.
7. Gather sort data from 10-20 participants.
8. Organize the dimensions that you obtained from all the participants to see if there are any common themes.

Strengths and Weaknesses:

+      Simple procedure.

+      Useful for understanding “user-derived” dimensions that should be considered when evaluating a product or service.

-       Analyzing the dimensions that emerge across participants requires some background in qualitative analysis.

References:

Curran, M.J., Rugg, G., & Corr, S. (2005). Attitudes to expert systems: a card sort study.
The Foot, December 2005, 15(4), pp. 190-19

Upchurch, L., Rugg, G. & Kitchenham, B. (2001). Using card sorts to elicit Web page quality attributes. IEEE Software, 84-89.

---

What’s Next in the Series?

The next UX method posting will describe, “reverse card sorting”, a variation on closed card sorting that focuses on hierarchical structures like tree menus and Web navigation. 

 

Method 7 of 100 - Claims Analysis

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

 by Chauncey Wilson

This is the 7th in a series of 100 short articles about UX design and evaluation methods. Today’s method is claims analysis. Claims analysis is a simple method for documenting the impact of new features or different designs on product usability and user experience.

A claim is a hypothesis about the positive and negative consequences for users and other stakeholders of a design decision. Claims can be based on theories, principles, patterns, guidelines or practical experience (Carroll & Rosson, 1992; Carroll, 1995;). Claims for new products are based on possible scenarios or stories of use and document the product team’s assumptions about how a set of features affects users or other stakeholders.

For example, a decision about whether to disable or hide entries in a pop-up menu carries positive and negative consequences for different stakeholders. Here is a brief and simple set of claims – positive and negative consequences – for a pop-up menu design that hides rather than disables menu items. Note that the claims (3 positive and 2 negative) refer to different groups – new users, expert users, developers, and product managers (consistent across products).

Design Feature: Pop-up menu items that are not available in a particular context are removed (rather than disabled)

+ The menu is shorter and theoretically more efficient for expert, high-frequency users.

+ There is less clutter in the pop-up menu.

+ The code for pop-ups supports the removal of items that aren’t available.

- The menu will have reduced positional predictability – a problem for new and non-expert users.

- Other products in our portfolio disable items in pop-up menus so this design feature would be inconsistent with other products and frustrate users.

One approach that we’ve used at Autodesk for claims analysis is to issue members of a product team red (negative consequences) and green (positive consequences) Post-it® notes and ask them to write their claims for different designs and scenarios of use as shown below. We ask people to identify the class of stakeholder (e.g., developer, new user, expert users, disabled user, product marketing) on each note so the perspective behind the claim is clear. The use of the two colors, a sort of visual claims analysis, can give you a quick sense whether a particular design feature is feasible – too many negative claims (red stickies) might portend problems. Here is a conceptual example where it is clear that Approach 1 may not be the approach that should be considered.

Method 7 of 100: Claims Analysis

When to Use:

Claims analysis is useful when:

• You are considering (debating) different solutions to a design problem.
• You want to get at the assumptions behind particular design.
• You want to understand what impact particular design decisions have on the user experience for one or more classes of users.
• You want to try a new approach to a design problem that is not well understood.
• You want to involve the entire product team.

Procedure:

1. Consider a list of the key stakeholders: new users, expert users, designers, developers, product managers, sales, etc.
2. Develop a set of scenarios that describe how people will use a product.  
3. Extract key features from user scenarios that will have an important effect on your users. For example, if you were planning on using a new type of control to find objects in huge lists (e.g., an elliptical browser), you might want to do a claims analysis on this feature.
4. Prioritize your list of key features.
5. Generate positive and negative claims for high-priority key features from the perspective of the stakeholders you listed in Step 1. You can do this in several ways:
   1. Brainstorming
   2. Interviews with users and other stakeholders
   3. Reviewing literature and reviews on similar features in other products
   4. Using a set of questions that help evoke possible consequences
   5. Using the “Yes, but…” format where a feature is described and you say, “Yes, but would it work for users who are disabled?”
6. Take the most important key feature from your list of claims and consider design trade-offs that will decrease the impact of negative claims and increase the impact of positive claims.
7. Repeat this process for the next key feature.

Strengths and Weaknesses:

+Simple and low budget.

+Multiple stakeholders can bring up consequences that might not be obvious. 

+Highlights positive consequences as well as negative ones.

+Makes assumptions of product team members explicit.

-Deciding which design issues to consider for claims analysis is a craft since there are thousands of decisions and hundreds of scenarios involved in creating even simple products.

-Claims analysis does not provide specific solutions.-Requires mentoring of product teams on design principles, patterns, and cognitive theories.

References:

Carroll, J. M. (1995). Introduction: the scenario perspective on system development. In J. M. Carroll (Ed.) Scenario-based design: envisioning work and technology in system development (pp. 1-18). New York: John Wiley & Sons, Inc.

Carroll, J. M. & Rosson, M. (1992). Getting around the task-artifact cycle: how to make claims and design by scenario. ACM Transactions of Information Systems, 10, 181-212.

Inspection Methods – Claims Analysis. http://insidethefactory.typepad.com/my_weblog/2009/09/

Usability First Glossary – Claims analysis. http://www.usabilityfirst.com/glossary/claims-analysis/

What’s Next in the Series?
The next UX method posting will describe “Repeat Card Sorting”, a technique for eliciting underlying dimensions or distinctions among a set of items.

 

Method 6 of 100 – Future Workshop

100 User Experience (UX) Design and Evaluation Methods for Your Toolkit

 by Chauncey Wilson

This is the 6th in a series of 100 short articles about UX design and evaluation methods. Today’s method is the future workshop. The future workshop is an approach that was originally proposed as a way to involve stakeholders in democratic decisions about public projects or social problems (Jungk & Müllert, 1987). In this context, “stakeholders” include anyone who is affected by a project, policy, product, or process – users, designers, developers, owners, neighbors, business partners, suppliers, lawyers, government officials, etc.

The future workshop method has been used in software design to develop visions of the future. For example, you could envision the “smart home of the future” or “home entertainment in the year 2020 or “the future of the UX field”. In the commercial setting, product team members can use the future workshop to develop 3, 5, and 10-year plans that take into account existing constraints and innovative ideas.  

Method 6 of 100: Future Workshop

The basic steps of a future workshop are:

• Identify the common problems or issues with a particular product, process, or system.
• Generate visions about how a future might be realized. This is called the fantasy phase.
• Discuss and analyze the feasibility of the ideas and solutions that emerged during the fantasy phase.
• Develop a future strategy based on various future visions, barriers, and constraints.

When to Use:

The future workshop is useful when:

• You are asked to develop a long-term strategy or vision of the future.
• You need to involve a large number of stakeholders in decisions.
• You want your immediate team and other stakeholders to recognize the constraints and barriers for implementing solutions.

Procedure:

There are four general phases in a future workshop:

1. Preparatory Phase
2. Critique Phase
3. Fantasy Phase
4. Implementation Phase

Here are the general procedures for each step. In a future workshop, facilitators will apply a wide variety of ideation and evaluation methods depending on the makeup of the group, the emotion that is associated with the problem, and the complexity of the problem.

Preparatory Phase – Planning, ground rules, and schedule

1. Define the goal of the workshop and a clear statement of the problem or issue.
2. Develop a list of stakeholders who will be affected by the topic of the future workshop.
3. Develop the ground rules and schedule for the future workshop and invite the stakeholders.
4. Require some preparation on the part of the participants. An important requirement is that your participants must engage in some preparation to develop a common understanding of the topic and ideas for the workshop. For example, if you were asked to create a future vision for architectural design software, you might ask people to read journal articles, look at some vision videos from innovative companies, or sketch out “science fiction” ideas before the meeting.

Critique Phase – Critique of the current system

1. State the goal of the workshop and the ground rules and schedule for the day.
2. Introduce all the stakeholders and consider some short warm-up exercises to develop rapport.
3. Critique the stated problem using brainstorming techniques. You could, for example, ask your participants to write problems on Post-It® notes and then stick them on the wall (good if you have quiet people who may not feel comfortable shouting out problems).
4. Organize the items from the critique into groups or piles based on similarity.
5. Create small groups of 3-5 people and ask them to take sets of problems and reformulate them into concise critiques of the current system.

Fantasy Phase – Unconstrained visions of the future

1. Conduct a fantasy session where each small group of participants verbalizes their desires, dreams, and fantasies to generate ideas about the future. You can many approaches here. For example you could take the negative statements from the Critique Phase and turn those into positive statements (see my blog on reverse brainstorming). You could also ask groups to write “science fiction” scenarios, use metaphor brainstorming, or draw pictures to envision unconstrained futures.
2. Have the small groups present outlines of the future systems to the entire group.

Implementation Phase

1. Conduct an analysis of the ideas from the fantasy session to determine their feasibility and to suggest ways to overcome any barriers that might hinder implementation of those ideas.
2. Outline a strategy about how you can overcome barriers and constraints.
3. Create an implementation plan that includes the tasks, people, and schedules you need to get to your vision of the future.

Strengths and Weaknesses:

+ Good for getting diverse stakeholders knowledgeable about key issues and involved in the decision-making process.

+ Uses a variety of common methods like brainstorming, affinity diagramming, and scenarios.

- Requires strong facilitation.

- Scheduling a diverse and large set of stakeholders can be expensive.

- Hard to do with remote audiences (you have to break it up some)

References:

Jungk, R, & Müllert, N. (1987). Future workshops: How to create desirable futures. London: Institute for Social Inventions.

What’s Next in the Series?

The next UX method posting will describe “Claims Analysis”, a technique for evaluating the negative and positive consequences of design decisions.