We are developing the instructions for use (IFU) for an imaging device used by physicians. The client decided to conduct a small, internal evaluation of a new safety feature of the device, and simultaneously evaluate the current draft of the IFU. They did this by giving the device and IFU to a few volunteer physicians to see if they could successfully use the device on a mannikin with no other training or instruction. Without realizing the impact, the client told participants to “do whatever they would do at work” during this evaluation.
What happened?
Most of the physicians read the IFU and chose not to use the new safety feature at all (which would be off-label use). The client wanted to understand if the IFU was sufficient on its own for overall use of the device and if users understood the new safety feature; they found that participants did not want to use the feature and it was unclear if the instructions were effective. They were able to gain some broad insights, but unable to answer either of their primary questions.
What can be learned?
In order to obtain actionable information at this stage in development, both objectives could not be simultaneously evaluated. The client needed to first determine if the feature was viable. Rather than giving the participants the device and IFU and telling them to use it however they wanted, the scenario needed to specify that they HAD to use the feature whether they wanted to or not so that we could understand:
Do they understand the purpose and when to use the new safety feature?
Can they use the feature safely, even if they don't want to?
It’s easy to want to test as many objectives as possible during early iterative usability testing. After all, it isn’t cheap to run multiple studies and you want to make the most of the opportunity. It sounds obvious, but when deciding how to set up scenarios for your participants during early research, the key is to focus on your main objective, accept that you can’t evaluate everything at once, and prioritize accordingly. The phrasing of the questions you ask can be the difference between information gathering and actionable insights.