by Karen Mason
I attended CHI in Vancouver a few weeks back. Authors can submit papers on original research to CHI. If your paper is accepted, you have 20-minutes to present it at the conference within a session where other papers and notes are presented.
One of my favorites was, "When Designing Questionnaires, Does it Hurt to be Positive?" [pdf], a paper presented by Jeff Sauro (and co-authored by Jim Lewis). It received the CHI Honorable Mention award.
It asks the question: Are both positive and negative items necessary in questionnaires? Great question. I’ve often thought that you needed to mix it up to get an unbiased, more balanced result. My intuition was that results will be biased towards the positive side if we don’t give subjects the option to respond with a lower rating.
“By providing a mix of both positive and negative items…, we are not predisposing our users to give us a positive response. We reduce the possibilities of biases: Acquiescent and Extreme response.”
- Acquiescent bias: Users have a tendency to just agree to whatever statements we have.
- Extreme response bias: User haphazardly circles the most favorable response for all.
Sauro & Lewis did a study where users were randomly given either version of these questionnaires:
- Standard System Usability Scale (SUS), which uses alternating positive and negative questions
- All Positive System Usability Scale, which uses all positive questions
It turns out you get virtually the same results, but when using the alternating positive and negative questions, your results are more prone to error. The 3 M’s:
- Misinterpretation: users interpret the question incorrectly
- Mistakes: user understand the question, but accidentally agree to the negative items when they meant to disagree
- Miscodes: researchers forget to alternate the items back
It is interesting to note that 18% of subjects who completed the Normal SUS responded favorably to both positive and negative questions.
- It’s OK to keep using Standard SUS for existing questionnaires, just be sure to check for errors.
- For new questionnaires, the recommendation is to use all positive questions. The negatives (3 M’s) outweigh the positives in alternating positive and negative items in usability questionnaires.
- Q. Did anyone try grouping positive and negative questions separately, vs. alternating positive and negative?
- A. Jeff answered that a user might still forget to switch mindset when the questions switched to negative. And more importantly, it doesn’t matter, so just use all positive.
Jeff has also posted a video of their presentation on YouTube.