Formative usability testing is great for identifying problems in an interface. By watching users interact with a system, you note when they encounter difficulties. For example, you might observe a participant searching in the wrong area for an item, or not adding an item to a shopping cart. Where traditional usability testing often falls short, however, is determining why those problems are occurring. Unless you specifically probe on what the user is thinking at that exact moment, you can lose the opportunity for accurate diagnosis.

Enter diagnostic testing. At Centralis, we use this method to uncover the “why” behind usability issues. In a diagnostic session, you not only observe behaviors, you determine what’s causing them. How does diagnostic testing do this? Read on:
 

1. Discussion about the interface is in-the-moment, not retrospective.
In a diagnostic usability test, the moderator is in the same room as the participant throughout the duration of the test session. As in many usability testing variants, the participant thinks aloud as he or she performs tasks using the interface. But instead of waiting for the participant to complete tasks before discussing thoughts and reactions (the retrospective technique style), the moderator asks questions during task completion attempts.

With a retrospective technique, participants have already moved through an interface and formed an opinion of how it works. Once seen, a screen cannot be “unseen.” For example, in the middle of a complex registration task, a user may think he has completed the process when he’s only on step 3 of 8. To diagnose why the user has reached this conclusion, it’s important to discuss what interface elements contributed to this impression before the user realizes there are additional steps.
 
2. The moderator is free to question the user, selectively probing elements of the experience.
Skilled questioning is what differentiates diagnostic usability testing sessions from regular sessions. This is accomplished through careful observation and probing to uncover the nature of an issue. Spontaneous, in-the-moment questioning can help to more accurately diagnose why issues are occurring. Why? Users don’t always know when they have encountered problems. Self-reports after the fact can be inaccurate and incomplete. And, some users won’t acknowledge a problem occurred in the first place.

For example, if the moderator observes a user leaning in close to the screen, the moderator can ask, “I noticed you leaned toward the screen. Can you tell me about that?” The participant’s response—in the moment, not after the fact—is often revealing. It can range from, “The type is too small” to “I don’t understand the instructions” to “This chair is uncomfortable.” In such instances, if the moderator had not probed, the opportunity for diagnosis would have disappeared.

Selective probing works not only for non-verbal cues, like body language, but also applies to actions, utterances, errors, etc. With in-the-moment questioning, the moderator can unpack statements from a think-aloud protocol to uncover: What does the user currently understand or misunderstand?

 Actions: “Was this what you expected to happen?”
 Utterances: “You sighed... Can you tell me more about that?”
 Verbal Cues: “I heard you say ‘What?’”
 Errors:  “What were you hoping to find here?”

 3. “Perception Checks” can determine what a user is seeing or not seeing.
Another useful technique a moderator can employ during a diagnostic usability session is a Perception Check. If, at any moment during the session, the moderator believes the participant is not perceiving or attending to something on-screen, the moderator can minimize the screen and ask the user to describe or draw a picture of what they were just viewing. As with user responses to probing, Perception Check data can be illuminating. Again, this technique is only valuable in the moment, before a user moves to the next area of an interface.

Here’s my all-time favorite Perception Check rendering of any participant in a usability test, ever. When asked to draw by recall a complex homepage loaded with information for job seekers, here’s what the participant remembered:





















All he gained was “Professional Guy in Suit.” Time to re-think the homepage.

As stated above, the focus of diagnostic testing is observation plus in-the-moment questioning of user behavior. As with any user experience research method, diagnostic testing has its strengths and limitations and is best used in certain instances. If you are interested in specific performance measures like time on task or number of clicks, for example, diagnostic testing is not a good choice. This data would be corrupted by the participant’s time in discussion with the moderator. If this data is of interest, then it’s best to use summative testing, or an online, unmoderated approach.

Diagnostic testing is most effective when want you to find and fix problems as part of an iterative design process to make an interface more usable. Because you have uncovered the reasons behind why a user searched in the wrong area for an item, or failed to add an item to a shopping cart, you can make more informed recommendations to improve the design.