The phrase usability testing gets thrown around a lot. It sounds great when you're planning a project. If you say you'll do some usability testing, then people get a warm feeling about your project plan.
After discussing it with a few people, I've concluded that there are a lot of myths out there about usability testing. I'll outline all of them that I've either heard from someone or thought to myself.
First let me explain what it is.
No. I lied. That's a huge topic, and I'm going to bypass it here. Instead, I'll refer you to the best summary I've read on the subject:
Dumas and Redish, A Practical Guide to Usability Testing. It's a bit old, and there are likely better summaries out there, but linking it nicely lets me avoid explaining what usability testing is.
I will, however, explain what the goals of usability testing are:
The idea is to watch your users interact with your product (or something similar to your product) in a way that allows you to see how well the product works for the users. In addition to finding problems, usability testing also tries to gather data that the testers can use to figure out how the problems should be corrected.
That description is designed to shoot down several of the bigger myths that I've run into.
Myth: Usability testing gathers statistical evidence that you can use to make decisions about solving a usability problem.
Running a good usability test on a single user is expensive and time consuming. There's a lot of data that gets analyzed, and I've found it to be a heck of a lot more useful to get more data about a single user than it is to test multiple users.
You end up running tests on very small numbers of people. I generally spend a fair amount of time before and after a single test preparing the test and analyzing the results. The test will find some problems, but you correct those problems before you analyze how bad the problems were. If a problem is bad, then you're likely to run into it. If a problem is small, you're not likely to be bothered by it again. It's much better to just assume that all problems you run into are large enough that they should be solved.
All forms of usability testing that I've done follow this formula: Watch a single user use your product. For every problem that user runs into, figure out what caused the problem, and decide if and how you need to fix it.
All problems a user runs into are considered real problems until proven otherwise.
Myth: Usability testing requires a lot of fancy equipment.
I hear this sometimes after people read about what large software companies use when they do usability testing. I've never seen it, but I've heard rumors about rooms filled with hidden cameras, one-way mirrors, and eye tracking devices. It sounds like fun, but it's really not needed.
When I do some usability testing, I try to understand the user as well as possible. I want to know what they're thinking when they click the wrong menu item. I tend to be in their face a lot more than someone standing behind one-way mirrors would be, but, for a lot of problems, a user being in an artificial environment with a couple of engineers breathing down their neck isn't as bad of a problem as it sounds. Certainly, a lot can be done this way.
Myth: Any old user will work.
The background of your user has a huge impact on how they view your product.
In the early stages of testing, I like to use coworkers as much as possible. They're easy to sucker into these tests; the first few times you do it, they even think it's fun! However, the data you generate from these tests requires so much interpretation that you can quickly get to the point where it's not worth your time. (They're great for the early tests, though. If nothing else, they can help test your testing procedure!)
Myth: Usability testing has to take a long time.
Myth: Usability testing can be done very quickly.
These myths are different, but the answer is the same. Usability testing can take as long as you need it to. It's fairly difficult to anticipate how long you'll need, however. Unexpected usability problems frequently come up. They need to be fixed in code, and if you're not lucky, that might take time.
If the dialog box or web page that you're testing isn't all that important, then it's possible to run tests that will fail to catch smaller problems. This can greatly speed things up, and it is possible to test something quickly and get away with it.
One of my favorite easy test to run is one that tries to decide between a small number of completely different approaches. This can be quick, but it's also possible that none of the approaches tried works all that well. I haven't been great at predicting the time required for usability tests that I've done, and I claim that it's a problem with usability testing rather than my own shortcoming.
One of the more dangerous forms of this myth is the belief that a large, complex product can be tested comprehensively in an amount of time that management will be happy with. Usability testing of a significant amount of UI code is a major project, and it really needs to be done continuously over the entire lifecycle of the product.
Myth: There aren't any other myths.
As I find some more time, I'll come back and address some more issues that I've run into. I think I've gotten the biggest ones that I've seen, though.
