Usability Over Time: Longitudinal Research Studies

by Desirée Sy on May 7, 2009

Lessons learned over time

Sometimes you meet someone new, and you have a great conversation. But no matter how strong your connection, and how much you learn from this first meeting, you'll only get a glimpse of that life. It doesn't compare with what you'll learn about that person in months and weeks to come. User experience practitioners who neglect longitudinal research are missing the wealth of information that comes only after time.

User research focused on single experiences with a feature or workflow uncovers different problems and issues than longitudinal research.

User research of single user experience uncovers	Longitudinal research uncovers
Discoverability problems Initial learnability problems Blocking issues preventing initial engagement Initial preferences Problems that block work within a few hours of use Issues affecting shorter workflows that take minutes or hours to complete Unexpected uses of product within minutes or hours of exposure	Awkward access to frequently used features Late-stage learnability problems Blocking issues preventing expert proficiency Preferences after sustained use Adoption and abandonment rates Problems that block work after sustained or intermittent use Issues affecting work that takes months or years to complete Unexpected uses of product that are only exposed over time

In short, what longitudinal research captures that research of a single experience cannot is the change over time—whether this change is in usage behaviour, comfort, perceptions, engagement, or productivity.

I participated in the "Best Practices in Longitudinal Research" workshop at CHI 2009 with a small group of academics and practitioners from five countries. We presented brief case studies, gathered over 30 issues to consider, and then shared our experiences to begin to examine about half a dozen questions. Here's a brief sampling of a few topics from the day's discussions.

The inevitable artifact of a gathering of UX researchers: flip charts with sticky notes.

What research attributes to hold constant, and what can change in a longitudinal study

To measure change meaningfully, at least one research attribute must be constant over the period of study. The fixed attributes most commonly used by workshop participants were the design of the product and a defined set of user characteristics (such as product knowledge, domain expertise, geography, age, and so on). User characteristics remained unchanged either by using exactly the same users for the whole study, or by rotating a panel of users with the same characteristics through the course of the investigation. For example, one study investigated how a group of users navigated Microsoft Word and Adobe Acrobat documents over a period of several months, while another determined the rate of learning of some users over several sessions for pointing device tasks.

I was in a smaller group of people who had studied the effect of design modifications on users' interaction with a product over time in their real world environment. I explained that the fixed attributes for our studies are the user characteristics and the design goals for the product, since the designs are changing over the course of the study.

Research methods best suited to longitudinal studies

Many different methods were used to gather data, determined by the research questions:

• diary studies
• usage logs and clickstream/instrumented data analysis
• periodic field ethnography
• periodic interviewing (both on site and remote)
• periodic usability testing (both on site and remote)
• retrospectives.

No one method was best suited to longitudinal studies. In fact, it was clear from some of the stories shared that for a given study, it's important to triangulate data collection from several methods. For example, in one study usage logs showed that users had not actually used the product as they described in their journals. Other case studies confirmed that usage logs alone cannot supply the context of use needed to understand workflow patterns.

I was interested in hearing from other participants of their success in observing longitudinal sessions remotely. Seeing participants' faces, not just their monitors, improves these studies by connecting participants to researchers. One company sent webcams to study participants as both a research tool and an incentive.

Future directions

As is often the case with workshops, the group uncovered more questions than we answered. Some directions for future work include:

• ongoing collection of case studies
• meta-analysis of different longitudinal studies
• creating a pattern library of user experience metrics
  (including what type of research each is suited for).

The day of discussions reminded me not to neglect the long-term relationships we have with our users, and to continue to plan user research over time. What lessons have you learned in your longitudinal studies?

Comments

[Geography Dissertation]

Blogs are so informative where we get lots of information on any topic. Nice job keep it up!!

[user experience research]

I am using Autodesk for my designs. Its really good.Nice article.