Design Values: Quality of Data over Ease of Data Collection

by Lynn Miller on November 21, 2009

This is a follow-up article to John Schrag’s post on Design Values. The original post described the values we believe contribute to a healthy software design practice.  This post looks deeper into why we value Quality of Data over Ease of Data Collection.

The first point that needs stating is that valuing quality of data over ease of data collection does not mean that we do not value Ease of Data Collection. It is important to make data collection as easy as possible in order to reduce the time commitment and financial costs, but cost reduction must not overshadow the need to use the right method for the data needed.

Let me give you an example.  A while ago, one of our product teams was trying to decide if they should continue to provide printed documentation to our customers.  The team created a survey which was sent out to the customer base. Surveys are low-cost and fast to create. At the same time, members of the product design team were on a string of customer site visits.  Customer visits are slow and fairly expensive. While at the sites, we looked around for the printed manuals or asked to see them. The results from the survey showed that 80% of our customers said they were using the printed documentation. The results from the customer visits showed that the majority of printed manuals were still in their original shrink wrap, locked up where end-users couldn’t access them, or were sitting in pristine condition on a shelf away from the users’ work area. The survey might have been low-cost and quick – but it got the wrong answer.  If we had continued to print the manuals based on the survey results, we would have spent far more money than the cost of the trips to the customer sites.

Getting quality data means matching the type of data collection to the type of data needed, and some types of data are harder to collect than others.

Gathering behavioral data seems to be most affected by the desire for easier data collection. Good quality behavioral data needs direct observation. Going for anything less than direct observation is sacrificing quality for ease of collection because of the factors that affect people when they are asked to self report their behavior.^1,2  When people are relying on memory, their answers can be distorted, incomplete, or biased.³

Surveys are the easiest form of data collection and thus the most abused. How often have you been asked to fill in a survey and found that you couldn’t answer a question and there was no way to indicate this? Sometimes it is just poorly worded questions, but often it is because the information should be gathered using a different technique.

For example, a well-known store sent me a survey a few weeks after I had made a purchase there. It had this question on it:  How many minutes did you stand in the check-out line? How could I possibly remember that with any degree of accuracy? If the store was considering making changes based on the data they were gathering, they would have been better off sending people into the stores to time the check-out line. Otherwise they might make changes based on my saying I stood in line for an hour, when really it only felt that way.

Quality data is data that allows you to make the right decisions. To get that data, UX professionals should be willing to put in the amount of work necessary and not be tempted to “get what they can” due to time or resource constrictions. Going for fast or cheap now can lead to expensive changes later on.

1 Blair, Edward and Scot Burton (1987). Cognitive processes used by survey respondents to answer behavioral frequency questions. Journal of Consumer Research 14: 280-288.
2 Bradburn, Norman M., Lance J. Rips and Steven K. Shevell (1987). Answering autobiographical questions: The impact of memory and inference on surveys. Science 236: 157(5).
3 Schacter, D. L. (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience. American Psychology, 54, 182-203

Comments

[Steve Johnson]

You appear to be making an assumption here that the quality of the data from the survey was inferior to that of the on-site visits. However, that doesn't have to be the case.

If the sites visited were mostly "important" ones (i.e. larger sites with IT departments who jealously guard the materials Autodesk permits them to have), then that's a very biased sample indeed and the survey is likely to be much closer to the truth than the site visits.

You also need to consider cause and effect. In recent years, the default has been for the customer to not receive manuals, so of course users you visit are not going to have them around. That doesn't mean they don't want them and wouldn't use them if they had them.

There's a subtext behind that 80% result. My guess is that the majority of customers don't want you to stop sending out manuals. They understood that if they said in your survey they weren't using the documentation that it would be used as an excuse to stop sending them out. So the survey, while maybe not reflecting a true answer to the question, did reflect a true indication of customer sentiment. All totally unscientific guesswork, but I bet I'm not too far off with it.

Of course, site visits to "important" sites (with which existing relationships may exist and where many users can be seen at once) is just another example of easier data collection.

I agree completely about the quality of survey questions; unfortunately, Autodesk surveys in particular seem to be very prone to this.

[Steve]

Sucks being part of the minority that actually uses the printed manuals though.

[Lynn Miller]

It is extremely important to use people from your target audience, but there are ways of using proxies to good effect.

Desiree Sy posted an article on just that:

http://dux.typepad.com/dux/2009/08/usability-testing-with-user-proxies-when-is-close-close-enough.html

[Brian Hoffman]

Excellent points Lynn! This is a battle I've had to fight many times before and still struggle to find a balance with. For example, when conducting usability testing, how important is it to do live testing with people from your target audience, which I believe gives you the highest quality of data, versus testing people off the street or remote testing?