by Desirée Sy on August 14, 2009
How can we designers get valid feedback from more design iterations in less time? One bottleneck in the design flow is finding a steady stream of usability testers. Between the extremes of the perfect (an actual user, on site) and the unacceptable (the developer who's coding the feature), lies the grey zone of user proxies. Can you use internal employees with relevant domain knowledge to usability test your products, and still get valid data?
The answer, as with all design questions: It depends.
We have successfully used internal employees as usability test participants on products where we have an agile user-centered design methodology (see “Adapting Usability Investigations for Agile User-Centered Design” in the Journal of Usability Studies for a lot more detail on our agile methods). Since in agile design the prototypes are incrementally and progressively getting closer to the end product, we took the approach of using tasks and testers that also incrementally and progressively got closer to real work done by actual users.
With agile, our designs are being iterated in an incremental series of “mini designs,” or design chunks, so the earliest prototypes do not allow people to complete full workflows, but only separate operation-level functionality. It is extremely hard to usability test these early “mini prototypes” externally because they do too little, have very little task-level context, and require extensive intervention from a facilitator to operate.
An example of one such early prototype might test the dragging algorithm for a zoom operation (where users zoom towards the click-point as opposed to the centre of the screen). While this interaction requires a high-fidelity prototype to test it, it's also possible to create low-fidelity prototypes to test granular interactions.
We usability test these types of “mini prototypes” internally, with extremely artificial forced tasks. For example, to test a brush resizing algorithm for a 2D paint application, I asked internal testers to match the brush size of different brushstrokes I'd prepared on a canvas using different prototypes. I switched between different prototypes for the testers since I was not testing discoverability of the function, but instead the smoothness and efficacy of the different algorithms.
Recruiting internal testers requires just as much thought as recruiting externally. You need to understand what you are testing and select testers accordingly.
We recruit internal testers from Quality Assurance, the Help Desk (Support), and our pool of subject matter experts. We do not use developers or other UX designers. All internal testers must work on a different product than the one being tested to minimize bias. Also, all internal testers must have particular qualities in common with our target users: in the brush resizing example, they needed to be people who sketched (so, for example, a QA person who was an expert in video rotoscoping but did not sketch was not recruited – just as an external person would not be recruited without hand sketching experience).
Our sessions with internal users are very fast, not conducted in a lab, and not recorded. Internal staff don’t want videos of them failing a task to be seen by people who might mistakenly see this as a sign of domain incompetence instead of a design failure.
Once the foundational interactions are locked down, we combine prototypes to enable more complete interactions that relate to a user activity. These more complete “workflow prototypes” are tested with external users. Initially, these users may be other user proxies (for example, students studying animation, industrial design, or other sketch-intensive fields who may not necessarily be users of the product). Eventually, we'll test refined workflow prototypes with actual users.
With some thought, I believe you can use internal employees and other user proxies successfully as part of an overall design validation strategy, whether or not you are using an agile development method. The main things to consider are:
- your users' specialized characteristics or environmental contexts
- how different these are from those of your user proxies
- how relevant particular characteristics or contexts are to using a given prototype.
Of course, if what you are testing is heavily dependent on a particular characteristic or context, then you need to usability test with real folks doing actual work—sometimes in the field. But more often than not, there are design decisions that don't depend on all user characteristics. By using internal employees and other user proxies to usability test interactions in early design work, you can increase your design velocity without sacrificing the validity of the feedback.
Subscribe