by Lynn Miller on November 21, 2009
This is a follow-up article to John Schrag’s post on Design Values. The original post described the values we believe contribute to a healthy software design practice. This post looks deeper into why we value Quality of Data over Ease of Data Collection.
The first point that needs stating is that valuing quality of data over ease of data collection does not mean that we do not value Ease of Data Collection. It is important to make data collection as easy as possible in order to reduce the time commitment and financial costs, but cost reduction must not overshadow the need to use the right method for the data needed.
Let me give you an example. A while ago, one of our product teams was trying to decide if they should continue to provide printed documentation to our customers. The team created a survey which was sent out to the customer base. Surveys are low-cost and fast to create. At the same time, members of the product design team were on a string of customer site visits. Customer visits are slow and fairly expensive. While at the sites, we looked around for the printed manuals or asked to see them. The results from the survey showed that 80% of our customers said they were using the printed documentation. The results from the customer visits showed that the majority of printed manuals were still in their original shrink wrap, locked up where end-users couldn’t access them, or were sitting in pristine condition on a shelf away from the users’ work area. The survey might have been low-cost and quick – but it got the wrong answer. If we had continued to print the manuals based on the survey results, we would have spent far more money than the cost of the trips to the customer sites.
Getting quality data means matching the type of data collection to the type of data needed, and some types of data are harder to collect than others.
Gathering behavioral data seems to be most affected by the desire for easier data collection. Good quality behavioral data needs direct observation. Going for anything less than direct observation is sacrificing quality for ease of collection because of the factors that affect people when they are asked to self report their behavior.1,2 When people are relying on memory, their answers can be distorted, incomplete, or biased.3
Surveys are the easiest form of data collection and thus the most abused. How often have you been asked to fill in a survey and found that you couldn’t answer a question and there was no way to indicate this? Sometimes it is just poorly worded questions, but often it is because the information should be gathered using a different technique.
For example, a well-known store sent me a survey a few weeks after I had made a purchase there. It had this question on it: How many minutes did you stand in the check-out line? How could I possibly remember that with any degree of accuracy? If the store was considering making changes based on the data they were gathering, they would have been better off sending people into the stores to time the check-out line. Otherwise they might make changes based on my saying I stood in line for an hour, when really it only felt that way.
Quality data is data that allows you to make the right decisions. To get that data, UX professionals should be willing to put in the amount of work necessary and not be tempted to “get what they can” due to time or resource constrictions. Going for fast or cheap now can lead to expensive changes later on.
1 Blair, Edward and Scot Burton (1987). Cognitive processes used by survey respondents to answer behavioral frequency questions. Journal of Consumer Research 14: 280-288.
2 Bradburn, Norman M., Lance J. Rips and Steven K. Shevell (1987). Answering autobiographical questions: The impact of memory and inference on surveys. Science 236: 157(5).
3 Schacter, D. L. (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience. American Psychology, 54, 182-203