When a design is giving users difficulty in a usability test, a common response of the observers is to dismiss the participants as non-representative:

“Oh, these aren’t our target users. Our users wouldn’t have this much trouble.”

It’s hard for observers to watch participants struggle, especially if they are the business owners or designers of the interface being tested. Sometimes it’s easier to dismiss the participants as unqualified than to acknowledge the issues with the design. As researchers, we need to help observers overcome this reaction and see the data accurately; otherwise, usability issues will not be fixed and the product will be less successful.

The first step in ensuring that usability findings are not summarily dismissed is to make sure that the participants are in fact your target users. Even if you’ve written a screener and hired a professional recruiter, it’s still possible for unqualified participants to slip into your study. Here are some tips for improving the buy-in of usability findings by ensuring your participant sample is unassailable:

1. Create clear, unambiguous definitions for each of your screening criteria.
Let’s say you want active online shoppers in your study. What does “active” mean? What makes someone an “online shopper”? Be specific in your screening questions to ensure you’re hitting the right target:

Ambiguous question:
How often do you shop online?
( ) Frequently
( ) Sometimes
( ) Never

Unambiguous question:
How often do you purchase items on websites?
( ) Every day
( ) 2-3 times a week
( ) 2-3 times a month
( ) Once or twice a year
( ) I never purchase items online

The first question leaves it to the participant to decide what “frequently” or “sometimes” mean. The second question enables you to eliminate participants who indicate they purchase items on websites less frequently than your desired threshold.

2. Ask critical questions from multiple angles.
Using more than one question to address important screening criteria decreases the risk that an unqualified participant will pass the screener. To build on the earlier example, if you want to recruit active online shoppers you can include additional questions to confirm and extend their previous answers:

Begin with:
How often do you purchase items on websites?
( ) Every day
( ) 2-3 times a week
( ) 2-3 times a month
( ) Once or twice a year
( ) I never purchase items online

And also ask:
Which of the following items have you purchased online in the past month?
( ) Books
( ) Clothing
( ) Household appliances
( ) Music
( ) Other (please specify_______________)
( ) I haven’t purchased anything online in the past month

And/or:
From which of the following sites have you made a purchase in the past month?
( ) Amazon.com
( ) BN.com
( ) Shopping.com
( ) Target.com
( ) Other (please specify _______________)
( ) I haven’t purchased anything online in the past month

And also:
Which of the following best describes your attitude about shopping online?
( ) Online shopping is completely safe
( ) Online shopping can be risky; I only purchase from sites I know
( ) I’m not comfortable shopping online

Asking similar questions from different angles and incorporating additional specifics provides a more comprehensive picture of potential participants’ behavior, which helps you more accurately determine whether they are truly representative users.

3. Avoid giving away the “right” answer in your question.
People agree to be called about research studies because they want to participate and earn the cash incentive. Even if they’re trying to answer honestly, it’s human instinct for potential participants to try to guess which answers will qualify them for the study and use that to their advantage. Multiple choice questions and distracters can conceal the responses that qualify users for participation, increasing the likelihood that they answer truthfully:

Easy-to-guess question:
Are you planning to buy a dishwasher in the next six months?
( ) Yes
( ) No

Difficult-to-guess question:
Which of the following appliances are you planning to purchase in the next six months?
( ) Cooktop
( ) Dishwasher
( ) Microwave
( ) Range
( ) Refrigerator
( ) None of the above

4. Ensure that people are comfortable answering truthfully by phrasing all choices positively.
People may be less comfortable being honest if they feel it will reflect poorly on them. Avoid using judgmental language to ensure that people won’t lie to save face.

Negative example:
What is your educational background?
( ) I never graduated from high school but I earned a GED
( ) I graduated from high school
( ) I started college but didn’t graduate
( ) I graduated from college
( ) I started graduate school but didn’t finish
( ) I have a graduate-level degree

Positive example:
What is your highest level of education?
( ) High school graduate/GED
( ) Some college
( ) College degree
( ) Some graduate school
( ) Graduate degree

5. Allow for gray areas by using scale ratings.
When asking people about their attitudes, it’s difficult to get a consistent answer if the only choices are “yes” and “no”. People change their minds; a participant may answer the question one way when responding to the screener, but another way in the context of the test session. Scale ratings allow for more variability, which enables participants to describe their opinions more precisely and consistently.

Less precise question:
Would you be willing to make an online purchase totaling over $1,000?
( ) Yes
( ) No

More precise question:
On a scale of 1 to 10, with 1 being unwilling and 10 being highly willing, how willing would you be to make an online purchase totaling over $1,000?

1     2      3      4      5      6      7      8      9     10

6. Recruit articulate participants that observers will respect.
Even though all types of users encounter usability problems, it’s more difficult for observers in denial to dismiss a participant who can clearly describe the nature of his or her confusion and engage in a meaningful dialogue with the moderator. Ensure your participants will have weight with the observers by including an articulation screen. Some examples include:

What is your favorite hobby, and why do you enjoy that activity?

If you could have dinner with any public figure, who would it be and why?

The purpose of these questions is not to evaluate the specific content of their answers (unless you’re studying hobbies or celebrities) - the goal is to select thoughtful participants who are willing to share their point of view. Instruct the recruiter to eliminate participants who provide one word answers or seem uninterested in engaging the question.

These screening techniques will go a long way toward ensuring that your study participants do in fact represent your target audience. Combining a well written screener with a top-notch recruiter is the key to a good participant sample. How do you find a great recruiter? Stay tuned for Part 2…