Usability Validation: Compliant with IEC 62366 and FDA

The usability validation is a test with objective means, whether the specified users can reach the specified user objectives (intended use) effectively and efficiently. 

Regulatory requirements for the usability validation

usability validation

The Medical Device Directive MDD explicitly calls that manufacturers must identify and control risks that arise from a specific context of use and by the characteristics of the user (for example level of education, intellectual and linguistic skills). The FDA does the same.

In order to demonstrate that these requirements are satisfied, it is necessary to validate usability. The IEC 62366 and the Human Factors Engineering Guidance Document of the FDA provide guidance on how this usability validation must be carried out.

1. Characterisation of the user

Thus, the usability validation must be carried out with typical representatives of the specified user group, usually in the form of participant observation. This test of course, assumes that you have specified:

  • The user group, for example, based on age, education, experience with the product, physical and intellectual abilities.
  • The context of use, such as "mental workload", the environment (temperature, humidity, brightness, ...) the tasks that are to be carried out and many more
  • User goals, the (medical) purpose is usually referenced here.

That users and the usage environment must be specified, is a requirement of the IEC 62366 in Section 5.1.

2. Involvement of representative users in the usability validation

Only for the usability validation will you need necessarily representative users and representative context of use (usage environment).

3. Number of subjects in the usability validation

The IEC 62366 demands representative users in the validation of serviceability. How many of these there are, however, it does not say. This is hardly doable, the number also depends on how homogeneous the different user groups are.

The FDA, however, references in its Guidance Document "Applying Human Factors and Usability Engineering to Optimise Medical Device Design" a study that mentions more concrete numbers.

Click here to enlarge

As you can see, you can find, on average, 85% of the utilisation problems with five representative users, with ten you can already 95%. Those are concrete figures!

4. Usability validation plan

According to IEC 62366, in the usability validation plan the main operating functions and especially since IEC 62366:2015 the safety-related usage scenarios must be included. Read more about this in the next chapter. The validation plan must also determine the approval criteria for the usability validation. The IEC 62366 gives example such as the following, that by the way can also be found in similar form in many usability files- if there is any. 

There will be some examples that X% of users in maximum Y minutes must have finished a particular task. Why only X% of them and not all? The fact that not all of them can accomplish the task is more of a hint for usability problem. And why in Y minutes? Where did this number come from?

The achievement of time goal is still nothing for the risk management. Because it could also mean that an acceptable but a non-achievement device (from the perspective of risk management!!) will be developed. The other way around, it could also be that in the process of maintaining targets itself remains an unacceptable risk.  

Lack of Usability leads to damages only through a chain of causes. The FDA will also not comment on usability tagets, but targets that affect the risk. With the usability engineering process you will be (hopefully) aware of potential risks. And this will then be examined in risk management. 

In the Seminar „Usability, Requirements and IEC 62366“ with Thomas Geis, you will be able to determine reasonable usability targets and quickly create usability files. 

 

Usability verification: inspection

The verification of the suitability for use will be successful with an inspection. This is a verification procedure whereby one or more experts examine your products on

  • whether the specified requirements, such as those found formulated in style guides and standards are implemented and / or
  • whether the product (in principle) is able to meet the usage requirements. If the usage requirement specifies that the physician must identify the patients with a low hemoglobin, and the system does not display the hemoglobin value, then this requirement is not met.

There are several methods for verification, such as the Cognitive Walkthroughs.

Usability Validation: Participatory observation

However, the inspection alone is not enough. You have to run through the core tasks within a participatory observation of genuine users in an actual or simulated user environment and let the security functions run. Only when users are actually able to achieve the targets, and thus prove that the user requirements are met, will the usability of your product be validated.

This assumes that you know all the core tasks. This will lead you to the frequently used functions. Next you need to know all the safety-critical functions. These result from the risk analysis. 

Methods for usability validation

There are several methods for testing the usability. A distinction is made between the following:

Usability verification: inspection

The verification of the suitability for use will be successful with an inspection. This is a verification procedure whereby one or more experts examine your products on

  • whether the specified requirements, such as those found formulated in style guides and standards are implemented and / or
  • whether the product (in principle) is able to meet the usage requirements. If the usage requirement specifies that the physician must identify the patients with a low hemoglobin, and the system does not display the hemoglobin value, then this requirement is not met.

There are several methods for verification, such as the Cognitive Walkthroughs.

Usability Validation: Participatory observation

However, the inspection alone is not enough. You have to run through the core tasks within a participatory observation of genuine users in an actual or simulated user environment and let the security functions run. Only when users are actually able to achieve the targets, and thus prove that the user requirements are met, will the usability of your product be validated.

This assumes that you know all the core tasks. This will lead you to the frequently used functions. Next you need to know all the safety-critical functions. These result from the risk analysis. 

User surveys

And so why do they still require the user surveys? When it comes to the purposes of IEC 62366 they don’t actually. Use questionnaires to compare products or development statuses quantitatively. And use interviews to objectify complaints.

Formative and summative evaluation of usability

What the FDA has already implemented, will be included in the second edition of IEC 62366 collection: The terms verification and validation of serviceability will be "replaced" by the terms formative and summative evaluation. But is this really a replacement?

Actually, the two pairs of concepts have little to do with each other - apart from the fact that they are all related to the testing of usability. The two pairs of terms refer rather to different dimensions: usability validation and usability verification differ in the objective of the test. The terms formative and summative evaluation differ in the time of the test.

For example, examining a GUI prototype during development would be a validation and a formative evaluation. A check of serviceability of the finished product with representative users in a representative environment of use would also be a validation, but a summative evaluation.

What you should not confuse the usability validation with

"Classic" system / product validation versus usability validation

You will find the difference between these processes described in another article.

Usability validation versus clinical trials

I came across an article in the "European Medical Device Technology" on serviceability. In it, the author points to a paragraph of IEC 60601-1-6, which strikes me as very important:

It should be noted that clinical investigations conducted according to ISO 14155-1 and usability testing for verification or validation according to this standard are two fundamentally different activities and should not be confused.

It is exactly like that! In validating the suitability for use, it is believed that it is generally possible to meet the medical purpose with the products. That it is assumed that the clinical trial was successful.

On the other hand, the validation of serviceability checks if, the representatives of a specified user group in a specific usage context actually achieve this specified result. Because the fact that the device is generally able to do this, does not allow conclusions about the interaction of the user with the product.

In other words:

  • The usability study has to validate towards the goal, if the specified user in the specified context of use can effectively and efficiently achieve the objectives. A prerequisite for the validation of usability are representative users.
  • Clinical studies, however, have to prove the goal that the product has a positive benefit-risk ratio. Patients are prerequisite for a clinical trial.

Although both studies are possibly carried out in the clinic "in real terms", the objectives are nevertheless different, though not overlapping. It is often useful to separate the tests of one another.

Usability validation versus field test

One of my favourite clients had the idea of validating his software in a field test, in order to meet the relevant legal requirements for validation. He almost committed an offense.

His thought to test, whether the purpose and usage requirements are met  in the field was really good. At the end of the day, the IEC 62366 demands that one involves "real" users . But to test the application in a field test in an uncontrolled way, corresponds to placing it on the market in an uncontrolled way. That is a criminal.

The correct approach would have been to carry out this test either in a simulated environment like a usability lab or in a strictly defined clinical trial. But at the clinical trials some conditions are met.

But especially for software requirements this is often not required. For this reason I recommend the Usability Lab. Anyone who has observed the manufacturer, how they view the users working with their software in a concerned, unbelievable and stunned way, will no longer suspect the following: That a usability lab is suitable for identifying usability issues. There’s no faster and cheaper way to experience, what many do not even really want to know when it comes to validating: The naked and often brutal truth about their product.

Further information

"Usability, Requirements & IEC 62366" Seminar

How to plan and execute such a validation (in the form of participant observation) you will learn in the seminar "Usability, requirements and IEC 62366" with my usability guru Thomas Geis. It will change your view of your own products radically. You will learn to understand why

  • there is always friction between your development department and product marketing,
  • Your customers seem to constantly have new desires,
  • Your company is doing harder than Apple,
  • You are not yet a millionaire.

However, you’ll not only learn to understand the causes, but above all, know the way to precisely overcome these difficulties. For me the seminar was and is groundbreaking.

If you want to be there, I suggest you  secure an appointment for next year. I could only secure Thomas Geis three times to come to Konstanz. Take advantage of this rare opportunity.

 

Usability Lab

We also have a usability lab, where we test customer products and demonstrate various methods. From Cognitive Walk-through to the expert evaluation. From eye-tracking to software-based usability testing (usability validation).

We have validated different products and recognised, which of these applications have partly catastrophic defects.

Links to the subject usability

  1. The Federal Institute for Occupational Safety and Health has published under the participation of notified bodies the ergonomics compendium "application Ergonomic rules and testing the usability of products.
  2. From the FDA environment comes the ANSI / AAMI HE75: 2009 - Human Factors Engineering - Design of Medical Devices

Both documents do not focus on software. But they contain specific provisions on the specification of the usability of products - in contrast to IEC 62366, which is a standard process. And another thing the two documents have in common: a volume of several hundred pages.

 

Author: Immanuel Bader

Author:

Prof. Dr. Christian Johner

Starter-Kit_rot_dunkel

A quick overview: Our

Starter-Kit

Learn More Pfeil_weiß
blog_rot_dunkel

Always up to date: Our

Institutejournal

Learn More Pfeil_grau