3 Ways to Get Feedback from Expert Users

Some customers are experts within a specific domain.  For example, an air traffic controller uses specific hardware and software to do their job because of industry standards and government regulations.  Doctors and nurses use specific instruments in surgery, different software for insurance, and specialized hardware and software to monitor patients.

Specialists will have different demands for their attention.  Split seconds decisions are made by pilots, engineers, doctors, armed forces, call centers, and more.  Within any industry, you can find deep domain experience that your customers have built up over the years.

So, let’s review three ways you can construct very effective usability tests for experts.

In the Teachback method, an expert explains concepts, workflows, or procedures to a non-expert (e.g.  a customer or a designer). Then, the non-expert tries to teach back what the expert has explained. After the Teachback session, the expert corrects any misunderstandings, which can be used to improve the design of the site or app.

I like to use this approach with decision support systems, call centers, and training sessions. It is interesting to see what the teacher says is important, then to listen for what the student says what was not clear, missing, or important to them.  Also, you will want to see what the student does.  Finally, listen to how the teacher re-directs the students and corrects their own instructions and actions.

Teachback is a technique for extracting information from experts that would be useful for gathering requirements, learning about workflow, and understanding mental models. The method is also used in healthcare to determine if patients understand instructions about medication or medical procedures. In the healthcare arena a medical provider, the nurse or doctor describes a procedure to a patient and then ask them to “tell me in your own words what I just said.”

The Teach Back method requires a domain or subject matter expert, a non-expert (the “learner”), and, optionally, a facilitator who takes notes and handles the recording. The learner and the facilitator can be the same person.

1. A subject matter expert describes a concept, task, or something else about a particular site or app. A facilitator records the sessions with audio or video (or just takes notes if electronic recording is not possible).
2. The non-expert (which might be a designer) listens carefully to the expert.
3. The non-expert is then asked to “teach back to the expert” what the expert has previously described or explained.
4. When the non-expert is teaching back what she learned, the expert corrects any misinterpretations, errors, or simplifications.
5. The session is repeated with another expert who again teaches the non-expert.  Use a different expert to get a new perspective.  Avoid the facilitator effect, where non-experts want to please their facilitator.
6. Review the sessions for common misinterpretations and errors.  You may be able to uncover potential design opportunities.

Ben Shneiderman created the User Race method during his time at the Palo Alto Research Center (PARC). In this method, expert users (with secure egos) compete against the clock and other experts to accomplish a specific goal. The results of the race are used to improve the quality of products and promote usability and good design.  Ironically, expert users are often neglected in typical usability tests.

In some respects, the User Race Method can resemble an episode of Iron Chef America (see picture).  The experts will watch what other team might be doing.  They will evaluate themselves against the other teams and the clock.  I like to use this method at User Conferences with expert users.  We create a scenario.  Then, we watch them work, listening for how they solve problems and collaborate.

You can use user interface races to:

• Reveal how experts work under pressure.
• Evaluate competitive products.
• Promote good usability in a public and entertaining way.
• Provide insights into shortcuts and tasks strategy employed by power users.

1. Choose participants who are open to public competition.
2. Choose a task for your race that is challenging for participants and engaging for viewers.
3. Develop the criteria for judging task performance. For most tasks, you might just use time, but you might also include a panel of judges to rate the quality of the solution (like the Olympic panels that judge divers and skaters).
4. Choose the judges for the competition and train them in the race method and ground rules.
5. Develop the tasks for the race. Keep the tasks challenging, but make sure they fit in a reasonable amount of time.
6. Develop the ground rules for local and remote viewers of the race.
7. If you are testing products that are new to the participants, give them equal amounts of time to “train for the race”.

1. Welcome viewers and participants to the race. Review the procedures and explain the task and scoring systems.
2. Have the participants read and sign consent and non-disclosure documents if they are from outside your company.
3. Provide time for the participants to warm up with the system if they are not using their own hardware and software.
4. Give the race participants the task and ask them to read and ask any questions. Avoid giving away any key tips.
5. When the participants are ready, give the signal to begin (Ready! Set! GO!)
6. When time is up or the participants have finished the task, Signal the end of the competition based on your rule for stopping.
7. Declare a winner and (optionally) award prizes (which could be gift certificates, plaques, or branded merchandise).

The Co-Discovery Method is an approach where two expert users attempt to perform tasks together while being observed. The participants help each other in the same manner as they would if they were working together to accomplish a common goal using the product.

You will want to encourage participants to talk out loud, while they are working on their task. This method is great for testing decision support system, collaboration tools, and systems with redundancies and protocols (such as weather or military).

1. Pair the test users into groups of two. It is preferable to pair two users who know each other into one group so that they won’t feel uncomfortable working together.
2. Provide the test users with the product to be tested (or a prototype of its interface) and a scenario of tasks to perform.
3. Ask them to perform the tasks using the product, and explain what they’re thinking about while working with the product’s interface.
4. Have them help each other in the same manner they would if they were working together to accomplish a common goal using the product.
5. Record the session.  Review and compile results to see underling patterns and design opportunities.

What do you think?  Do you have any methods or techniques for working with expert users?

Related articles

• Comparing User Task Flows with the CUTA Method
• 5 Methods to Collect Data with Diary Studies
• 3 Ways to Collect Data with 5-second Blink Tests
• See What Clicks with Visual Affordance Tests
• Join the UX Dallas Meetup Group