The Need for Realism
The FDA guidance released in February 2016, Applying Human Factors and Usability Engineering to Medical Devices, states that the conditions under which human factors (HF) validation testing is conducted should be “sufficiently realistic to represent conditions of actual use." This can be challenging, however. How do you conduct a "sufficiently realistic" usability test of a device that will be used under stressful conditions?
Some examples of stressful situations include:
giving someone an emergency EpiPen® injection;
using a defibrillator on an unresponsive person;
responding to a vascular access emergency with a home hemodialysis machine; and
experiencing anaphylaxis yourself while trying to use a rescue medication.
It’s important to start with as much realism in the test environment and task scenarios as you can. For the EpiPen® and defibrillator examples, this would mean having a mannequin on the floor, as if a person had just collapsed. To be even more realistic, you could hire standardized patient actors and coach them to display the appropriate signs and symptoms of the disease state. You could also simulate a busy, chaotic scene with a group of actors playing concerned citizens. In OR environments, you can play a recording of surgical equipment sounds, intercom noise, and alarms to add realism. In some cases, it may even be possible to conduct the usability test in the actual environment. A colleague of mine recently tested EMTs using an emergency medication in an open field, for example.
Types of Stress
Consider the types of stress you expect users to experience when using the device. This may be mostly cognitive (such as being unable to focus and think clearly) or mostly physical (such as being unable to manipulate a device with high dexterity.) In life-and-death situations, both types of stress often occur simultaneously. Look to your device’s use specification and task analysis to identify the types of stress that might occur.
When people are stressed, the following can happen:
muscles tense up;
heart rate increases;
respiration rate increases;
blood vessels in the arms and legs dilate;
hands can shake;
pupils dilate, letting in more light;
short-term memory decreases, particularly verbal memory;
concentration decreases; and
it can be difficult to handle intellectual tasks and behaviors.
We need to find ways of inducing this during HF validation testing, while at the same time behaving ethically toward the participants.
There are techniques you can use to induce stressful conditions. Common ones include:
imposing time limits for task completion;
increasing the incentive for fast and accurate performance;
using multitasking to represent reduced cognitive function;
introducing distractions to simulate decreased concentration; and
introducing physical demands on the hands and arms.
The first technique, imposing time limits, is appropriate for most stressful situations since it accurately reflects a sense of urgency. The moderator directs participants to complete the scenario as quickly and accurately as they can and tells them that they will be timed. (Having a timer ticking loudly enhances the effect.) To increase the effect even further, you could offer a bonus incentive for participants to perform the scenario correctly within the time limit.
To simulate reduced cognitive function, you can require participants to multi-task as they perform scenarios. For example, you could show them random words (parrot, truck) on a screen and ask them to categorize the words (bird, vehicle) verbally while they quickly and accurately perform their tasks. Another possibility is to burden them with unnecessary information as they perform their tasks. A well-known example of this is the “digit span test” which requires people to memorize a series of numbers and then recall them.
Distractions can also introduce stress and divert attention at strategic times during the scenario. For example, the moderator or an actor can break in to ask a question, or the phone can ring and the participant needs to answer it.
Finally, if the stress is mostly physical rather than cognitive, you could ask participants to undergo a predecessor task in which they have to exert physical energy. This could be an activity involving the hands and arms, such as lifting and carrying boxes. You could ask them to move the boxes quickly and stack them perfectly, which might also elevate the heart and respiration rate. Afterward ask them to perform the task you intend to test.
Limitations of Simulation Techniques
Even when we try hard to accurately simulate stressful situations, the techniques we have aren’t perfect. It’s hard to know exactly how people will feel and react in a real emergency. It's also hard to know the degree to which our simulations approximate real-world scenarios. To address this, do some research ahead of time – talk to the target users about their biggest challenges, for example, or view online videos of people in the emergency situation. Finally, keep in mind that consequences of failing tasks in a usability test will never be the same as they would be in real life.
Of course, you should not impose excessive stress on test participants because it isn’t ethical. When designing a study where you will intentionally induce stress, have an institutional review board (IRB) or ethics committee review protocol to ensure that the well-being of participants is protected. And certainly, if participants show signs of excessive stress during the study, stop the scenario.
As human factors practitioners and device manufacturers, it is our responsibility to ensure that devices on the market are safe and effective for the intended users, use environments, and expected conditions of use. For devices used in stressful situations, the better we can simulate the conditions, the more likely it is that we will uncover design flaws – and correct them.
Zwillinger, B. (2018) Is Simulated Use Problematic for Usability Testing?
Leffingwell, T.G. (2017) Incorporating Stress in User Testing, in MDDI Online.
Wiklund, et al (2011) Usability Testing of Medical Devices. CRC Press.
Kahol, K. et al (2009) Cognitive Simulators for Medical Education and Training. Journal of Biomedical Informatics 42, 593-604.