Designing the User Experience at Autodesk

Nicely Done Steve Jobs

By Erin Bradner

In Spring 2010, there was an Apple bloom across Autodesk – iPads had sprouted up in conference rooms from San Francisco to Shanghai. Some appeared carefully wrapped in protective cases, some with diminutive keyboards. Many sat atop conference tables unadorned, barely weeks old, occupying the table space and mindshare previously occupied by Macbooks and PCs.

A second bloom happened later that spring when a new variety of iPhone… black, white and now glinting with stainless-steel…began dispersing into the iPhone ecosystem at Autodesk. And just last week I watched the Apple enthusiasts on our user experience team prairie dog across a bay of cubes to chatter excitedly about the iPhone 4 announcement. I also watched those same colleagues hush mournfully, the day Steve Jobs died.

This week I asked my colleagues to tell me how Steve Jobs, via Apple, has influenced design thinking at Autodesk. I’ve compiled some of the responses below. A common thread is that Jobs exerted influence on Autodesk through his courage to re-invent products and re-imagine entire industries. Designers and design leadership at Autodesk have drawn inspiration from his fanatical attention to detail, such as his ability to sell the minutiae of “cut, copy and paste” on the iPhone, and through his consistency in delivering powerful, beautiful technology. Here are their words:

 

“One comment of Steve Jobs that stuck with me was his advice to “Edit, always,” because while that is easy to say, it is inherently hard to do in business. Businesses are constantly bombarded by opportunities, competitive threats and customer demands and can naturally end up pursuing multiple priorities. Jobs strove to do a few things really well – a smaller set of products, a focused set of principles but then executing them completely.” – Amar Hanspal

"Steve Jobs was a believer of User Experience, not in the traditional practice of the trade but in its essence. He ultimately knew that building a great product was intertwined with the user – their needs and perceptions. He connected not just tasks, error rates, and usability but also emotion and passion to create a powerful legacy." – Julie Schiller

"Steve Jobs wanted to go beyond making products that are useful -- he wanted to make products that reach people on a deeper level, products that people fall in love with.  As someone who did 'fall' for Apple products, I know what that experience is like and strive to foster that feeling among users of Autodesk software. Steve Jobs (and his team at Apple) showed many UX professionals at Autodesk, myself included, what a good user experience feels like. This influences the user experience goals for our projects every day." – Edward De Guzman

“Unlike almost any other CEO out there, Steve Jobs had faith in the human experience as a business model. The experience of Apple’s design is what drove profit, not the other way around. He intuitively invested in something that no financial analyst could predict, but only measure afterwards.” – Alison Kather

 “Steve Jobs showed to the world how User Experience can make all the difference. His impact is profound as technology companies across the globe have changed their culture to make UX a priority over features and capabilities.” – Ananth Uggirala

“Steve Jobs and Apple’s designs are such a penultimate of achievement that anyone in a design oriented field cannot help but be inspired to follow suit. Their designs being centered on the individual, rather than the company or the project, have helped influence the next generation of design at Autodesk. When working on AutoCAD for Mac, Steve’s reality distortion field helped bring back excitement to an otherwise fairly mature brand.” - Rob Maguire

 

Autodesk is one of those tech companies whose design culture has evolved thanks to Steve Jobs. It is rare if a week goes by at Autodesk when one of Apple’s innovations isn’t used as a gold standard for user-friendly UI or loyalty-inspiring customer experience. Yesterday in a conference room filled with managers, interaction designers and researchers, we deconstructed the Genius Bar. The experience distilled down to three important elements:

1. It’s not a transaction, it’s a relationship: they don’t give you a number as you wait, they call your name instead.
2. They are empathetic: the Apple geniuses, not technicians, walk around a bar, not a desk, and stand next to you as they respectfully examine your dead phone or empathetically listen to your wireless networking woes.
3. They are empowered to exceed your expectations: they typically solve your problem plus give you something more, something unexpected… an iOS upgrade or even a new laptop.

The Genius Bar, iTunes, iPhone and Macbook each have their fanbase at Autodesk. This month AutoCAD for the Mac celebrates its 1-year birthday. Autodesk has a growing list of Mac and mobile apps for tinkers and seasoned professionals alike. We’ve both drawn inspiration from Apple and are delivering products in partnership with Apple. Our Mac and mobile products aren’t faultless. The products Steve Jobs helped create aren’t faultless either. And their user interfaces aren’t always as intuitive as some in our field might hope. Yet Apple products strike an emotional chord – they deliver experiences that are human, empathetic and that routinely exceed expectations. Nicely done, Steve Jobs.

Are you an Innovation Giant?

by Erin Bradner

At the annual ACM Human-Computer Interaction Conference (CHI 2011) this year, I approached an unsuspecting human-factors engineer from IBM and asked him, point blank: “Are you an innovation giant?”. I wanted to push on the idea I believed Bill Buxton was getting at when he shipped his personal collection of technology to the conference. This idea was that innovation isn’t fundamentally about giant personalities and unparalleled genius. It’s about wicked smart people with healthy curiosity and a lot of hard work. Innovation is not a singular flash of genius in an open ocean of possibility, it’s the outcome of cross-fertilization and mash ups in a technology Galapagos. 

 
Jim Lewis, from IBM, was the unsuspecting, wicked smart designer I cornered. In 1993 he helped to design the Simon, which has come to be known as the world’s first smart phone. He is an author on several of the user interface patents used in the iPhone today. Here’s the surprise: Jim tells me that there may have been genius behind the world’s first smartphone -- a genius who had the vision for the PDA-phone mash up and the emotional intelligence to motivate a brilliant team to execute on that vision. Let’s hear it from Jim.

Are you an innovation giant? If not, what do you do?  
Jim Lewis: Nope. Not a giant. I'm pretty close to the national average height :) I’m a human-factors engineer. I take published findings and methods of investigation and apply them to designs to make systems easier to use. I've done that at IBM since I walked through the door 30 years ago, right up until just before this conversation … applying what we know to try to make systems that are easier and more pleasant to use.
 
Why is the Simon included in the Buxton Collocation at CHI?
Jim Lewis: When I heard about the Buxton Collection, I knew it was something I had to see. I haven't seen a working Simon in over 10 years. At CHI, Bill Buxton asked if there was a genius designer on the team. That is a difficult question. For the most part I would say 'no' but if I had to name one person, I would credit the guy who started the whole thing off, Frank Canova. If anyone on the team was a genius, it was him. The Simon was his vision.

What was out there and known in the personal organizer market at the time? The HP 85, the Apple Newton and the Sharp Wizard were all fairly well known. It was his idea to take a cellular phone form-factor and work within that form-factor to provide the organizer functions. That’s what is now known as a smart phone. It was also Frank’s idea to make it a touch screen. He got a small team together at IBM to make a prototype in 1992 and he took it to COMDEX. The response at COMDEX was positive enough that they wanted to make a product out of it. In 1993 they came to my department with a question about the minimum size of usable touchscreen buttons. That’s when I joined the team.
 
Who was on the team?
Jim Lewis: Frank came to our group of graphic designers, industrial designers and human factors engineers. There was the graphic designer, Peter Eastwood, who had both technical and artistic skills. He designed icons and also designed a screen refresh process that was ultimately patented. There were industrial designers who designed the appearance of the device, they were all trained as artists. They made sure the device was consistent with the branding principals of the organization. And then me, the human-factors engineer who worked out the details of the interaction design. We truly worked as a team.
 
What does that mean, to work as a team?
Jim Lewis: We would paste the graphic designs all over the wall ...  “If I touch this, it goes to that.” We spent a lot of time and paper getting to where we could visualize the interactions. We asked ourselves “How are we going to design a keyboard for email, address book entries?”. This connected us to the initial question that brought the Simon developers to my department in the first place - “How small can I make a key on a touchpad and make it usable?”. Since I had been following touch screen research, I was familiar with the University of Maryland research by Ben Shneiderman and Andrew Sears. They had published findings showing success with designs using lift-off on buttons as small as 2mm square. A key characteristic of touchscreen buttons is when they trigger: do they act on land-on or lift-off? Land-on takes effect as soon as you touch the screen… that's fast. But the consequence of that is that there is no room to adjust if you make a mistake, so you need to use large buttons. Land-on is good for ATM kiosks.
 
For a small display with small buttons, you'll want to use lift-off. For example, if you have an onscreen keyboard with lift-off and  you touched the letter F when you meant G, you could slide over to G and then lift your finger off -- not the fastest possible input but more forgiving, and you can make the buttons much smaller. With well-designed lift off, you can acquire very small targets. You can slide your finger in a text string, get the insertion point where you want it, listen or watch for feedback such as a beep, then drag to select text. This is an example of how I do my work. I keep track of what is being published and understand how that may apply to a specific design problem. In applying research to design, you can misapply research when there are variables you're not considering. It makes for a fascinating and satisfying intellectual exercise.
 
To quote the Buxton Collection notes from CHI: “First shown in 1993, this was the world’s first so-called “smart phone”. It only had two buttons: on/off and volume. Access to all other capabilities was via a resistive film touch sensor … Not bad for 14 years before the iPhone!” Did the CHI blurb get it right? When you were working on it, did it feel like the Simon was a decade ahead of its time?
Jim Lewis: From the perspective of the touchscreen yes, it felt ahead. At the same time, we knew that we were working on something that was not completely dissimilar from what was out there. There was no other device that would allow you to have someone on the phone, look up a contact and give them that information while you're on the phone. The device itself was unwieldy… it was a brick. It was paying the price for being early 1990s technology. It wasn’t particularly durable either. If you stepped on it, you could break it. Its price point was pretty high too - $899 in 1993. It also didn't have small multi-function hard buttons with the touchscreen soft buttons. We called it a personal communicator to distinguish it from PDAs (Personal Digital Assistants). PDAs had the organizer tools, but none were able to act as a regular telephone. You’d see people carrying phones and PDAs and it seemed logical to say “let's do a mash up!” (although we certainly didn’t use that term in 1993).
 
What did it feel like while you were working on that team? Was there any feeling of uncertainty?
Jim Lewis: I don't remember a feeling of uncertainty. We, the full team, built the entire operating system from the ground up so all of the aspects of touch were completely under our control from a very low level. My main memory of it was that it was the most delightful design experience I've ever had. The team was outstanding, in terms of skills and the relationships we had with each other. Those relationships have carried on. The Simon didn't survive in the market due to price and fragility. But those of us who worked on it still get together and have lunch. Julie Goodwin did the coding for the phone. She still sends out occasional emails to get us all together for lunch.
 
Are there any Simon patents that you’re especially proud of?
Jim Lewis: When you look at the Simon patents it looks like cast of thousands. But as I recall, there were about forty people who were full-time on the project (I’m not sure about the exact number, so please don’t hold me to that). Marking a phone number then popping up a contextual menu which includes the option to dial is in one patent. We were proud about that. It was clever. We also did some things around panning and zooming images received by fax, but no multi-touch – that technology wasn’t generally available yet. The graphic designer received a patent for his screen refresh technique.
 
Tell me about the design process. Was it innovative?
Jim Lewis: We followed a fairly straightforward user-centered design process. Once we had the key concepts and the primary tasks, we looked at how people did those tasks using current products. I had no formal design training. The graphic and industrial designers had formal design training. When the graphic designer made icons, I would run usability tests to be sure they were usable.
 
We wanted to avoid the mistakes of current products. We wanted to move ourselves forward in the design space we'd chosen. Other than trying to create something and using iterative feedback from usability tests to sharpen our designs, nothing fundamentally different about our process comes to mind.
 
Does it take genius mind and giant personality to innovate? If not, how is it done?
Jim Lewis: My feeling is there are many paths toward innovation. I wouldn't exclude the possibility that someone has a brilliant idea and all the skills needed to make it a successful product. Only a relatively small proportion of innovation comes from that source. If you ran a company, would you bank on that small proportion? Not if you wanted to stay in business. The probability isn't there.
 
The more common path is assembling a group.  It doesn't hurt if you have a little bit of charismatic leadership there, which is what we had with Frank Canova. The hardware engineers were following the best engineering practices; the software engineers were following the best software practices. Then Frank brought industrial design, graphic design and human factors onto the team. Does a team always come together in a magical way? Probably not, but it sure did for the Simon. We contributed our best practices and, in this particular case, it turned out well for everyone involved.
 
That said, Frank was clearly the leader and visionary. He made everyone feel great about the project, no matter what stage things were in. He had a knack about making you feel really good about being part of it. When you had to put in those extra hours, you felt you were part of something larger. You were willing to go the extra mile. In some ways it was unusual. Everyone was at the top of their game. I've worked on a lot of different projects at IBM with good colleagues, but with that team, we were all friends. We spent a lot of our time laughing… that’s what happens when you make a team out of people with a lot of sharp intellect and sense of humor.

~ 

So the story behind one of the most wildly popular innovations of the past 20 years, the world’s first smartphone, includes genius and magic after all. The genius came from spotting a mash up, assembling a cross-functional team and motivating that team with vision and compassion. And the magic? Jim tells us that the magic came from the human factors – not just human-factors engineering, but the full spectrum of human factors that operate when the right team comes together at the right time.

A Technology Galapagos at CHI 2011

by Erin Bradner

I was at a conference last month and found myself in a room teaming with people engrossed in technology. Two people were using smart phones, others were tapping on PDAs, and one was using a word processor. None of this is blogworthy until I tell you that the transfixing technologies in the room were all digital antiques curated in a makeshift gadget gallery. On display was the world's first portable word processor from 1978 with only six keys on its keyboard. Another curious artifact was an early touchscreen PDA featuring buttons designed to be played like a trumpet. But what drew me in was the world’s first touchscreen smart phone, a dinosaur that predated the Apple iPhone by 14 years. That phone, designed by IBM, is the origin of several user interface patents in the iPhone today. 

Image from the Buxton Collection Sampler. Image Source CHI 2011

This anachronistic smattering of digital curios was part of a private collection shipped to the Human-Computer Interaction Conference (CHI 2011) in Vancouver, B.C. The owner and curator of the collection was the inexhaustible gadget hound, Bill Buxton. Throughout the conference Buxton gave non-stop, guided tours of his collection for anyone who walked by. Think Darwin in the Galapagos. Meandering through the collection he would point out the design quirks, elegance and ingenuity of each device. I noticed that he rarely got two gadgets into a thought without referring to a larger innovation taxonomy that seemed self-evident to him. While walking and talking, he would grab two dissimilar devices from opposite ends of the room, hold them next to each other and explain that they were in the same family; one was simply the miniaturization of the other. Next he would scoop up a specimen and explain how it was the cross-fertilization of species from the PDA genus, the tablet genus and a trumpet.

Over time, as Buxton spoke, I noticed that his rapid-fire audit of the distinguishing traits of each gadget included complimentary terms like wonderful, inventive, and inspired. Yet never, while I was there, did I hear the words big idea, giant or genius. Then, standing somewhere between a Reverse Polish Notation calculator and a foot-operated mouse, he blurted out a Darwinian truth. I can’t quote, so this is the gist:

That’s how we do it. As designers, we slice up a given design space along the various dimensions of size, speed, permutation, etc… we amplify and attenuate, expand and contract until we find something interesting and suspiciously doable.

 Something clicked for me then. Perhaps the collection of digital doodads catalogued before me wasn’t as it appeared to be. It was not a systematic progression of gadgetry punctuated by paradigm-shifting sparks of the Genius Inventor. It was a mashup of form factor, features and user interface. It was the work of legions of tireless designers actively seeking under-populated niches in a design space and growing ideas there – then cross-fertilizing and mutating and growing more and more ideas.

Economic or ergonomic, it also struck me that some form of natural selection had to be in play.

Freshly liberated from the cult of the Genius Inventor, I went downstairs to lunch. As luck would have it I found myself sitting across the table from Jim Lewis, the Human Factors Engineer from IBM who designed the user interface for a phone I held in my hand only minutes before in the gadget Galapagos upstairs. I felt compelled to confront Jim immediately with a pressing question: “If you’re not a genius, if you’re not a giant, if the UI for the world’s first smart phone didn’t spring fully formed as a big idea, then how did you design it?” I let him finish his lunch and then I asked. I’ll recount that conversation in my next DUX post.

 

 

Recommending Net Promoter

by Erin Bradner

I have just one quick question for you:

"Would you recommend this blog post?"

Wait, I’m ahead of myself. Net Promoter fever is getting the best of me. Technically, I should let you read this blog post before I collect the one number I need to grow my readership.

Perhaps you've already been exposed to Net Promoter fever? The symptoms are:

• A compulsion to ask, "Would you recommend this product?"
• A maniacal classification of the respondents into three categories: promoters, neutrals and detractors
• Followed by an irresistible urge to subtract the percent of promoters from the percent of detractors to get your Net Promoter Score

Last month, my colleagues and I caught the fever. It all began when we used the Net Promoter method to analyze user satisfaction with a feature in our product.

We chose Net Promoter because we wanted more than an average satisfaction score. We wanted to understand how a specific feature, the L&T feature, factored into our customers’ total product experience. Through an approach called key driver analysis - frequently used with Net Promoter - we identified the experience areas that inspire customers to actively promote our product.

How and what did we measure? We launched a survey aimed at measuring customer satisfaction with discoverability, ease-of-use and relevance of the L&T feature.

We also included rating questions on overall product quality, product value and product ease-of-use. We then calculated mean satisfaction scores for each experience variable. Satisfaction is plotted along the x-axis in the chart below. Next we ran correlations between each satisfaction score and the question:

"Would you recommend this product?"

Those correlations are plotted against the y-axis in the chart below. We call the y-axis Importance because correlation to the question, "Would you recommend this product?" is what tells us how important each experience variable is to our customers. Moreover, it tells us what is important in a way that is meaningful to both our business and our customers.

The gist of it is that no one is going to recommend a product without really liking it. When we recommend something, especially in a professional setting, we put our reputations on the line. Recommending a product is admitting we're way beyond satisfied. It signifies that we are even willing to do a little marketing & promotion on behalf of this product. This altruistic, highly credible and free promotion from enthusiastic customers is what makes the recommend question meaningful to your business, and to businesses like Autodesk. Promoters are going to encourage others to purchase our product and are likely to re-purchase. Great products that ignite promoters are the products that drive the growth of our business.

I'm in the software design group, not in the executive suite, but I had to slip in the term driving growth to decode the method I referred to earlier as key driver analysis. This is because plotting satisfaction for the L&T feature against willingness to recommend showed us that the L&T feature is lower on the y-axis relative to the other measures. Since we have determined that satisfaction with L&T is not as important as willingness to recommend; it is therefore not as important to driving growth of product sales. At least not when L&T is compared to measures higher on the y-axis like product quality and product value.

            Figure 1 - Some graph data is simulated

 

Keep in mind; this analysis is not inferring there isn’t any work to be done on the L&T feature. On the contrary, the quadrants on the graph tell us exactly which aspects of the L&T feature to work on next. If something popped up in the upper left quadrant, labeled FIX, that would be the highest priority area. I've heard this quadrant affectionately called the FIX IT YESTERDAY! quadrant. Nothing showed up here, thankfully.

Still, we plan to invest design and engineering time on the L&T feature so our next priority will come from the LEVERAGE or MAINTAIN quadrants. We'll pick L&T RELEVANCE since that's what showed up there. Discoverability and ease-of-use of the L&T feature are in the HOLD quadrant so we'll prioritize those last.

Figure 2 - Some graph data is simulated

 

Next we got curious how much the user experience contributes to a willingness to recommend the product. We knew from previous research the strongest predictors to willingness to recommend are helpful, responsive sales support (in our case, resellers), and substantial functionality at a good price (value). We ran multiple-regression and found that the variables for user experience contribute 36% to recommendation scores (n=2170). Value and reseller support accounted for another 18%. Surprised by the significant contribution of product experience variables, we ran another multiple regression on a different data set (n=1061) and found the contribution of user experience variables to be 40%.

Now let's get back to the Net Promoter score. Or…let’s not. It turns out that calculating the net promoter score isn't as important as graphing and using the key driver charts. By graphing the charts over time, we see how our investments in key areas pay out. We can watch features move from the FIX IT YESTERDAY! quadrant safely into the LEVERAGE quadrant. Inspiring more customers to promote our product is what motivates us. It's not about the score; it's about how we use it. Try it yourself.

“I highly recommend it.”

Small Sample, Big ROI

by Erin Bradner

As a user researcher in a company that practices user-center design, I rarely find myself justifying user research. Autodesk values design research because we consistently derive actionable design direction from it. Specifically, in the division I work in at Autodesk, user research was long ago justified and ROIed. To use Eric Schaffer’s term, user research has been institutionalized. What remains somewhat of a mystery among non-researchers at Autodesk is how we do what we do with usability results…how do we know we’ve found a legitimate problem with such a small sample of users?

I got that ‘how do you know it’s not noise’ question recently. We had just completed a usability study of the AutoCAD software installer. We had observed and interviewed 11 customers on the same tasks: finding the download on Autodesk.com, downloading, installing and launching the software. We found that the download button was misleading (it didn’t look clickable enough) and that some of the error messaging in the installer was cryptic (“ADR Not Empty.” Huh?).

Within the week, a colleague reported that she had already begun responding to the usability issues. Jumping on the highest-severity issue first, she and her team had redesigned the download button. And had used A/B testing against the design. Her testing showed that the redesign produced a 5% improvement in click-through rates; which in her words was "considerable" in light of the rigorous testing the download page had previously undergone. While our small-sample research showed only three out of our 11 users had experienced a problem with the download button, her statistically significant results from an A/B test immediately showed that the usability fix yielded considerable increase in click-throughs.

Shortly after results from the A/B testing was announced, a colleague from the web analytics team stopped me in the hall and asked: “How did you do it?” How did you detect what amounted to a statistically significant design flaw with just 11 users?

I gave my colleague the answer I’d given many times before: with small-sample usability research we report patterns not p-values. We use our experience and expertise to separate high-severity problems from low. My colleague seemed satisfied with my response. But I went back to my desk dissatisfied. It struck me that my standard response – it’s patterns not p-values – eschews the small-sample statistics that is the secret sauce behind usability.

I decided it was time to revise my standard response to give credit to small-sample statistics. Two minutes on Google turns up no shortage of discussion on the topic of small-sample usability research ranging from the academic to the polemical to the pragmatic. To summarize:

The time-tested minimum sample for a usability test is five users. Testing the same tasks with five users we’re not likely to see most problems, we are just likely to see most problems that affect 30% or more of users. With five users, we’ll detect approximately 85% of the problems in an interface, given that the probability a user would encounter a problem is about 30%. These numbers are based on the binomial distribution and probability statistics.

In practice, in my department at Autodesk, this means that when we’re testing with small samples:

• we pick up high high-frequency (30% or greater) issues, not low frequency issues.
• when we know the probability of picking up a specific problem that we want to shoot for, we calculate the number of users we need to test with.
• we put confidence intervals on our data when reasonable.
• we triangulate our findings with statistics from other data sources when available, such as feature usage data, web analytics and support calls.
• we strive to test between 5 and 20 users, iteratively, for each study.

In the case of the download button not looking like a button, we observed 3 of 11 users struggle. From this, we estimated the impact on all users. Calculating the probability and confidence interval, we were 95% confident that between 9% and 57%* of all users would fail to notice that the button is clickable.

Since writing this post, I have not yet been asked in the hallway ‘how do you do it.’ I can’t say yet if returning to my roots and quoting small-sample statistics will be a more satisfying response than ‘patterns not p-values’. Feel free to weigh in here!

*We used calculators and insights developed by Jeff Sauro at www.measuringusability.com.