Usability Testing: FAQs
Have questions about usability testing? We've assembled a list of the questions we're asked most often. Please feel free to contact us for more information.
What is usability testing?
Usability testing is a research technique that identifies and diagnoses difficulties people have using a product. By seeing what people actually do in a usability test, we can pinpoint sources of error, clarify areas of confusion and eliminate user frustration. Usability testing can be used to improve any type of user experience, including consumer products, services, websites, software applications and physical environments.
Back to top
When should I do usability testing?
It’s never too early or too late to do usability testing, but you learn different things by putting your product in front of users at different points in the development cycle. Testing early gives you the opportunity to respond to what you learn before a significant investment has been made in building the product, while testing later enables you to give users a more realistic sense of what the product will be like.
The questions you have about your users should drive your decision about when to do usability testing. When you want to know if certain features are valuable and desirable, test early in the concept stage, or consider using a focus group. If you’re primarily concerned with users’ ability to find information, test as soon as you have mock-ups. (A card sort might also be helpful.) If you want to study users’ reactions to your full brand experience, test after the design is complete. Of course, periodically testing your current products helps you identify and fix usability issues incrementally, between major redesigns.
Back to top
What if my product/site isn’t built yet?
You don’t need a finished product to conduct usability testing. In fact, testing with prototypes enables you to gather user input while it’s still early enough to incorporate it into the design. Prototypes can take a variety of forms, from foam product models to hand-drawn or digital sketches. Most often, we work with semi-functional prototypes that incorporate a limited set of features — just enough to give participants the illusion of completing key tasks. (We’ve developed our own method of creating digital prototypes; you can read about it in one of our presentations to the Usability Professionals’ Association.)
Back to top
Where can my testing take place?
Usability testing can be set up pretty much anywhere your users are available. Formal settings like usability labs or market research facilities are best if you want a controlled environment and expect to have many people from your team observe the sessions. (Check out Centralis Lab.) Testing can also take place on-site at your offices with our mobile lab if you have a quiet, out-of-the-way space for testing and a nearby conference room for the observers. We’ve also set up our mobile lab at annual meetings, conferences, training centers and user group meetings to take advantage of many users gathering together in a single location.
Another alternative is in-context testing, which involves going into the users’ actual setting and asking them to perform a series of tasks with their own equipment. (This type of testing is often combined with field studies.) In-context testing gives you the added benefit of seeing the environment in which your product is actually used, enabling you to assess the impact of factors like lighting, noise levels, temperature and the availability of other tools. (If you’ve ever tried to use your iPhone with gloves on, you can appreciate how important these factors can be.) In-context testing is valuable if you’re studying a product that is highly customized or touched by a series of users, such as manufacturing software or ERP systems.
Finally, remote testing is a possibility for incorporating feedback from worldwide users while minimizing travel costs. In our opinion, remote testing yields less data than face-to-face sessions, but it’s an acceptable option if in-person testing is not feasible.
Back to top
How many users should we include in testing?
We usually recommend recruiting 6-8 participants from each unique user group. This sample size will reveal most of your product’s usability flaws without overlearning, slowing down the process or breaking your budget. Most of our clients have two or three user groups, which equates to about 12-24 sessions, or two or three days of testing.
To find the right sample size for your study, consider the types of people who use your product to do different things. For example, doctors and patients would probably use a hospital’s web site differently, so you’d want feedback on their unique tasks and perspectives. Other common user types include experienced users versus new users, supervisors versus line staff, or customers versus employees.
Back to top
Will my results be statistically significant?
No — and that’s okay. Diagnostic usability testing is a qualitative technique; it provides a rich, detailed understanding of how people use a product. We rely on face validity to rate the importance of a problem: if we see users struggling with something, and we can understand why they’re struggling, we know we need to address the problem with a design change. While it is possible to reach statistical significance by using a larger sample to measure how many people in the overall population are likely to have the problem, we feel this requires a lot of additional time and money but provides minimal incremental benefit.
While large sample sizes are usually not required for usability testing, projectiblity is important for other types of research questions. If you’re interested in understanding who are the most frequent users of your product, or whether there’s a market opportunity for a new concept, you might want to consider conducting a survey instead.
Back to top
How do you find people to participate?
We’ll work closely with you to define the types of users you want to include in testing, and formalize those requirements in a screening questionnaire. We then contract directly with a recruiting agency to secure the participants. The recruiters either work from customer lists that you provide, or leverage their own databases of people who have agreed to be contacted about participating in research. All our recruiters meet our rigorous quality assurance standards, ensuring that you get the right types of people for your study.
In some circumstances, our clients prefer to recruit their own participants, particularly if we’re testing internal applications where the end users are employees. In those cases, we can consult with you on screening criteria and help schedule the sessions.
Back to top
How many tasks can we include?
The number of tasks that can be covered in a single usability session depends on how complex they are — for example, buying a book may be faster than completing an online job application. Based on what you want to learn from testing, we’ll recommend the number of tasks each session can accommodate. Often, our clients identify “high priority” scenarios that all participants complete, and have a set of less critical tasks at hand to work in as time permits.
Back to top
Who actually conducts the test sessions?
All of our projects are staffed by a team of two User Experience Specialists — we’re highly experienced professionals with graduate-level training in cognitive psychology, social science, human-computer interaction or related fields. Everyone at Centralis is both a researcher and a designer, which means we understand the practical implications of what we see in testing. Our recommendations are concrete and specific, and we work with your design team to translate the test findings into a better user experience.
Back to top
Can I observe the sessions?
Absolutely! It’s important for the design team and key stakeholders to observe usability testing first-hand. Our in-house lab accommodates up to 10 observers, or we can use our mobile lab to bring testing to you. Observers can follow along via web conference, although in-person is the best viewing experience.
Back to top
What is eye-tracking, and do I need it?
Eye-tracking equipment monitors a participant’s eye movements across a computer screen, recording the path of their gaze and points of fixation. The outputs of an eye-tracking study include “heat maps” that use color to indicate the screen areas with the greatest number of fixations, and “gaze plots” that depict eye movements across the screen.
Some usability professionals feel that eye-tracking augments conventional usability testing by providing additional data that participants can’t verbalize. Others (including us) have fundamental concerns about what heat maps and gaze plots actually tell us about a user’s experience. While eye-tracking is a method we have used, we feel it has inherent limitations.
Eye-tracking studies rest on the assumption that eye fixation is equivalent to seeing – if your eyes looked at it, you must have seen it. But human perception is not that simple. Sensation and perception are related but independent processes; if we processed everything our eyes fixated on, our brains would be overloaded. Heat maps run the risk of overstating the extent to which users attend to specific areas of the interface, because they only reflect the fixation, not the perception.
While eye-tracking may be useful in limited circumstances, we believe there are many less-invasive, lower-cost alternatives for understanding what users perceive.
Back to top
How quickly can we get the findings so we can move ahead with the design?
Most of our clients can’t afford to wait if they hope to integrate what they learn from usability testing into their design. On the day after testing, we’ll facilitate a collaborative workshop with your team - while everyone’s observations are fresh in their minds, we’ll document the findings and come to consensus about the key implications. Our full report will follow within two weeks, but thanks to the workshop, you’ll have a prioritized list of the most important usability issues and our initial recommendations for how to address them as soon as the day after testing.
Back to top