Usability Testing with User Proxies: When is "Close" Close Enough?

by Desirée Sy on August 14, 2009

How can we designers get valid feedback from more design iterations in less time? One bottleneck in the design flow is finding a steady stream of usability testers. Between the extremes of the perfect (an actual user, on site) and the unacceptable (the developer who's coding the feature), lies the grey zone of user proxies. Can you use internal employees with relevant domain knowledge to usability test your products, and still get valid data?

The answer, as with all design questions: It depends.

We have successfully used internal employees as usability test participants on products where we have an agile user-centered design methodology (see “Adapting Usability Investigations for Agile User-Centered Design” in the Journal of Usability Studies for a lot more detail on our agile methods). Since in agile design the prototypes are incrementally and progressively getting closer to the end product, we took the approach of using tasks and testers that also incrementally and progressively got closer to real work done by actual users.

With agile, our designs are being iterated in an incremental series of “mini designs,” or design chunks, so the earliest prototypes do not allow people to complete full workflows, but only separate operation-level functionality.  It is extremely hard to usability test these early “mini prototypes” externally because they do too little, have very little task-level context, and require extensive intervention from a facilitator to operate. 

An example of one such early prototype might test the dragging algorithm for a zoom operation (where users zoom towards the click-point as opposed to the centre of the screen). While this interaction requires a high-fidelity prototype to test it, it's also possible to create low-fidelity prototypes to test granular interactions.

We usability test these types of “mini prototypes” internally, with extremely artificial forced tasks. For example, to test a brush resizing algorithm for a 2D paint application, I asked internal testers to match the brush size of different brushstrokes I'd prepared on a canvas using different prototypes.  I switched between different prototypes for the testers since I was not testing discoverability of the function, but instead the smoothness and efficacy of the different algorithms. 

Recruiting internal testers requires just as much thought as recruiting externally. You need to understand what you are testing and select testers accordingly. 

We recruit internal testers from Quality Assurance, the Help Desk (Support), and our pool of subject matter experts.  We do not use developers or other UX designers.  All internal testers must work on a different product than the one being tested to minimize bias. Also, all internal testers must have particular qualities in common with our target users: in the brush resizing example, they needed to be people who sketched (so, for example, a QA person who was an expert in video rotoscoping but did not sketch was not recruited – just as an external person would not be recruited without hand sketching experience).

Our sessions with internal users are very fast, not conducted in a lab, and not recorded. Internal staff don’t want videos of them failing a task to be seen by people who might mistakenly see this as a sign of domain incompetence instead of a design failure.
 
Once the foundational interactions are locked down, we combine prototypes to enable more complete interactions that relate to a user activity. These more complete “workflow prototypes” are tested with external users. Initially, these users may be other user proxies (for example, students studying animation, industrial design, or other sketch-intensive fields who may not necessarily be users of the product). Eventually, we'll test refined workflow prototypes with actual users.

With some thought, I believe you can use internal employees and other user proxies successfully as part of an overall design validation strategy, whether or not you are using an agile development method.  The main things to consider are:

• your users' specialized characteristics or environmental contexts
• how different these are from those of your user proxies
• how relevant particular characteristics or contexts are to using a given prototype.

Of course, if what you are testing is heavily dependent on a particular characteristic or context, then you need to usability test with real folks doing actual work—sometimes in the field. But more often than not, there are design decisions that don't depend on all user characteristics. By using internal employees and other user proxies to usability test interactions in early design work, you can increase your design velocity without sacrificing the validity of the feedback.

Comments

[Andre]

I know you guys are busy, but in the hope of spurring more of these interesting posts, here is a thought on the time factor of high velocity usability testing.

Tools you use in a 30 minute test vs tools you use in a 30 month project.
Using Revit 2010 as an example:
File>New>Project vs File>New>Family
In any test of the software, you will need to use new>project and are unlikely to spend enough time in the program to use new>family so the interface gets skewed toward new>project. However, in the real world, you only use new>project once per project... maybe two or three times at the beginning as you work through false starts. Then you use new>family often at certain phases of the project. Again, testing at the wrong time and the choice doesn't get used at all, testing at another time and the tool gets used every day, several times a day. In Revit, either a flow chart or a bad test nested new>family below new>project adding clicks to a frequent process over the old interface which had them equal (technically the same number of clicks but in a slower interface it takes longer)

High velocity here doesn't get you accurate data... for a simple function which probably wouldn't warrant user testing.