Not The Usual Suspects: How To Recruit Usability Test Participants

When it comes to usability testing, too many of us resemble Captain Louis Renault at the end of Casablanca. He’s the one instructing his staff to “round up the usual suspects.” Sound familiar?

What follows are a few simple ways to round up a more interesting — and truly useful — cast of characters for your next investigation.

Finding the Right Participants

The first step is to figure out which user groups you want to hear feedback from. Analyze your content. Be clear about the audiences you’re trying to reach. Step back and really assess your goals:

• Who was the project designed for?
• Is there any special equipment, knowledge, or background necessary to appreciate it?
• How do you want the user to benefit from your site or application?

Assess what you know about your current site users. Use site statistics as well as customer feedback to figure out your typical audience. Is it primarily middle-aged, male, American? Are you content to reach that group, or would you like to shift the profile in some way? For example, would you like to attract younger, older, richer, poorer, more or less highly educated users?

In your testing, you’ll want to include people who fit both the current and desired profile of typical site users.

How Many People Do I Need?

Jakob Nielsen has popularized the rule of 5: the theory that 85% of a site’s problems can be found with as few as 5 participants.

It’s certainly true that you can learn a lot in just a few interviews. But it’s hard to get input that’s representative of all your various user groups with such a small number. I would recommend you expand your number of participants to around 15, if only to account for a few inevitable no-shows, and to cover your various user groups.

Finding the appropriate people can be a major hurdle to performing frequent and effective usability testing. Recruiting agency costs can be substantial: in 2003, Nielsen found that the average agency fee was $107 per participant. If that’s not in your budget, then you will have to do some outreach of your own.

Sampling Techniques

You will need enough users to fill your sample population with the correct mix
of experience, skills, and demographic characteristics. Usability test results are only valid if the participants reflect your current or desired profile of typical site users. Here are a few sampling techniques to ensure that you get a good mix of participants.

Random Sampling

Random sampling requires that you define a targeted population of users, then ensure that every member of that population has an equal chance of being chosen. It is considered the best way to get an unbiased, representative sample, because it ensures that you don’t just approach the people you know and like. The tricky part, of course, is defining your population. Random sampling can also be a time-consuming and expensive approach if you have a very large target group.

But if you’re testing a company’s intranet site, for example, then you have a well-defined population. You could put all employees’ names into a hat and draw out names until you have the desired number of participants. Voila: a random sample.

If you’re to test an existing ecommerce site, why not ask the marketing department for a list of current customers, and select randomly from that list?

Quota Sampling

You may already have used quota sampling even if you aren’t familiar with the term. It basically means that you grab whomever you can get, subject to certain quotas. For example, if you’re interviewing 20 people total, then perhaps no more than 10 should be men (depending on your current and desired user profiles).

Once you’ve reached your quota for any given factor, move on to the next: age, income, race, preferred browser, whatever. Although not truly random, this approach is widely used commercially and very common in usability testing.

Opportunity Sampling

Opportunity sampling is a popular technique because it is relatively easy — you simply position yourself somewhere you are likely to find users who fit your desired criteria, and take your sample from people who are available at the time. Although it has its flaws, opportunity sampling can be a very useful technique when you are trying to target a very specific group in a short period of time. Just be aware that this opportunistic approach is far from a random method, since people with strong opinions are usually more willing to co-operate. The same is true of people who have plenty of time on their hands — like kids, or adults who are unemployed — and therefore may not be the demographic you’re looking for.

But if you are interested in the opinions of economists, for example, it makes sense to position yourself in a place where you’re likely to run into economists – perhaps a university department building or an economics conference.

If you are interested in improving the accessibility of your site and need to find users with visual impairments, consider contacting rehabilitation agencies. They may be able to connect you with some of their clients, or perhaps an assistive technology instructor who would be happy to promote the cause of Website accessibility. Another resource to consider is your local university’s Office of Disability Services.

Snowball Sampling

Snowball sampling is a great way to expand your typical pool of participants. It’s often recommended as a tool to find populations who are harder to reach. If your desired characteristic is rare, it might be extremely difficult or time-consuming to find representative users through traditional methods.

Find one or two key people who fit the profiles you’re looking for and ask them to recommend people similar to them who would be willing to be interviewed. By moving steadily away from the people you know, you can reduce bias and expand your group of potential testers.

Snowball sampling is often used for exploratory, qualitative research, usually conducted through interviews. Therefore, it lends itself well to usability testing.

Enticements

Money

Money is everyone’s favorite motivator, of course. If you have the budget to pay participants for their time, then you’ll find people are much more willing to help you. Money is also a great time-saver. A temp agency can help you find users with the desired skills and background. But if there is no budget allocated for usability testing (often the case), never fear. You have other options.

Food

Homemade snacks go a long way toward convincing people that you appreciate their time. I recommend bar cookies – here’s a recipe for oatmeal chocolate-chip bars that contains so much butter it is practically foolproof.

School Credit

If your targeted user groups include students, see if you can convince a computer instructor to offer extra credit for participating in usability testing.

Gifts or Gift Certificates

Microsoft, for example, offers a variety of software programs to its usability testing participants. If your company or client sells products or services, consider giving your usability participants their choice of gift.

Thank You

A sincere note of thanks for users’ time is always appreciated. They will be especially glad if you know their boss’s name and make sure to cc him or her.

How to Approach Participants

• Introduce yourself. First, let them know who you are. Once they know your name, and that you represent a reputable company or client, they will be more inclined to help.
• Avoid using the word “test.” People get anxious about tests. It can make participants feel like they are being tested, not the site. Make it clear through your careful use of language that it is your site that is being evaluated, not their Web skills. Rather than “participate in usability testing,” use phrases like “try out our site” and “give us your feedback.” Let them know that they are the experts here.
• Reassure them about your privacy policies. Make it clear that no identifying information will be attached to user’s comments. This protects your participants and encourages honesty.

Privacy is particularly important if you are asking coworkers to test an intranet site. Employees need to be assured that their supervisors won’t learn about their criticisms of the site, both for their peace of mind and for the effectiveness of your usability testing. The goal is honest feedback.

Conclusion

The more usability test participants you can round up, the more likely you are to conduct frequent and effective tests of your Web projects. To that end, it’s worthwhile to think carefully about each project’s intended audience, hone your sampling and recruiting techniques, improve your spiel, and offer adequate compensation. With a little planning, you can easily round up more than just the usual suspects.

For more on this topic, see:

• http://www.psybox.com/web_dictionary/Samplingtechniques.htm
• http://www.ahref.com/guides/design/199806/0615jef4.html
• http://www.useit.com/alertbox/20030120.html

Frequently Asked Questions about Usability Test Participants