Worst-Case Scenarios for Usability Testing
Updated: 4 days ago
Background on the Term “Worst-Case”
I recently was asked for advice on selecting worst-case scenarios for usability testing of medical devices. Actually, the concept of “worst-case scenarios” is somewhat outdated; the term was included in the original version of IEC 62366:2007 but is not used in IEC 62366-1:2015 or IEC TR 62366-2:2016.
Annex A of the original 62366 said:
“The USABILITY SPECIFICATION describes two groups of USE SCENARIOS: those USE SCENARIOS that occur frequently and those that represent a reasonably foreseeable worst-case. In identifying those USE SCENARIOS the MANUFACTURER should focus on:
a) facilitating the specific tasks that the MANUFACTURER intends the USERS and the RESPONSIBLE ORGANIZATION to perform; and
b) minimizing the RISKS that can arise if the intended USER or the RESPONSIBLE ORGANIZATION try to perform other tasks not intended by the MANUFACTURER.”
In this context, “worst-case” seems to refer to times when users are misusing the device; that is, using it in a way that the manufacturer did not intend.
Regarding usability testing specifically, Annex D of the original 62366 said:
“Simulated clinical environments permit evaluation in a controlled manner in a setting containing some or all of the essential attributes of the actual clinical environment for which the MEDICAL DEVICE is being designed. Simulations facilitate creation of worst-case USE SCENARIOS and complex failures. A high-RISK MEDICAL DEVICE or one involving tasks that are more complex can be tested in high-fidelity simulators...”
In this context, “worst-case” is referring to use of the device under the most unfavorable expected conditions. This better aligns with the way the U.S. FDA uses the term; the FDA’s Guidance Applying Human Factors and Usability Engineering to Medical Devices (2016) says:
“Formative evaluation can be conducted with varying degrees of formality and sample sizes, depending on how much information is needed to inform device design, the complexity of the device and its use, the variability of the user population, or specific conditions of use (e.g., worst-case conditions).”
Keep in mind that these conditions of use may include the larger context, such as a lack of training or negative transfer from experience with similar devices.
Selecting Scenarios for Usability Testing
As the FDA says in the passage above, there is no prescriptive requirement for which scenarios to include in formative testing. The scenarios you choose should exercise aspects of the user interface you are designing, and/or represent a good range of scenarios to help better understand potential risks of any severity. That said, late stage formative studies can focus on the highest severity hazard-related use scenarios in preparation for human factors validation testing.
Human Factors Validation (a.k.a. Summative) Studies
When it comes to human factors validation testing, both IEC 62366-1 and the FDA Guidance are explicit: you need to include the hazard-related use scenarios.
IEC 62366-1 says:
“Select the HAZARD-RELATED USE SCENARIOS for SUMMATIVE EVALUATION
The MANUFACTURER shall select the HAZARD-RELATED USE SCENARIOS to be included in the SUMMATIVE EVALUATION.
The MANUFACTURER shall select either:
– all HAZARD-RELATED USE SCENARIOS; or
– the subset of the HAZARD-RELATED USE SCENARIOS based on the SEVERITY of the potential HARM that could be caused by USE ERROR (e.g. for which medical intervention would be needed).”
So, instead of selecting “worst-case” scenarios, the team should either include all the hazard-related use scenarios identified in the risk analysis, or a subset based on the severity of potential harm.
Selecting Conditions for Usability Testing
Because the conditions of use can influence the number and types of errors that users make, it’s important to simulate those conditions when conducting human factors validation studies.
The FDA says:
“Human factors validation testing is conducted to demonstrate that the device can be used by the intended users without serious use errors or problems, for the intended uses and under the expected use conditions…The test conditions are sufficiently realistic to represent actual conditions of use.”
In this sense, you may need to simulate “worst-case” environmental conditions, noise levels, lighting, or stress and time constraints. This is discussed in my previous blog, Simulating Stress in Usability Testing.
In summary, for formative studies select use scenarios that allow you to design the best user interface possible and uncover previously unknown risks, and simulate conditions of actual use when appropriate. For human factors validation testing include all hazard-related use scenarios or the ones with the highest potential for harm, and always simulate conditions of actual use.