Usability Testing in Times of Corona

Manufacturers cannot simply not carry out usability tests in the time of corona. Both IEC 62366-1 and the FDA insist on these tests. And there are no exceptions because of the coronavirus.

With the appropriate safety strategy, you can still comply with both the safety requirements and conduct legally compliant usability tests, thus fulfilling the prerequisite for the authorization of your devices. Even in the time of corona.

This article

• Describes an easy-to-implement safety strategy
• Gives you tips to help you create your own standard operating procedures quickly
• Includes a recruitment checklist for download. And, as a result, saves you a lot of work

Formative evaluation: involvement of test users necessary?

Whether the involvement of usability experts only in a formative evaluation is enough and test users are not required depends on factors including the following:

• Development status of the user interface
• Possibility of improving user interfaces where use errors are only found during the summative evaluation and the cost of doing so
• Availability of usability experts who can carry out heuristic evaluations, cognitive walkthroughs and other inspection procedures
• Regulatory requirements (e.g. MDR, IEC 62366-1, FDA)

Remote testing possible?

The decision as to whether it is possible to carry out remote usability tests should be based on the answers to these questions:

• Is the “remote location” sufficiently representative of the actual use environment?
• Is it even possible to bring the medical device to this location and operate it there? For example, are the power supply connections required available?
• Is it possible to adequately monitor the use of the device at the remote location? For example, is it possible to send sound and images?

b) Regulatory background

Irrespective of the market, the German Medical Devices Act (MPG) obliges manufacturers to identify and manage risks resulting from poor usability.

IEC 62366-1 does not prescribe a mandatory method for the formative evaluation. This means that even an assessment without the participation of representative users can conform with the standard.

In contrast, in its Human Factors Engineering Guidance, the FDA recommends the participation of “users”.

For summative evaluations, the regulations insist on “real usability tests” with test users.

In the next steps, we will explain how to carry out these usability tests safely and in compliance with the law in the time of corona.

Good ventilation of the rooms makes a significant contribution to minimizing the risk of infection, from aerosols in particular.

b) Technical requirements

The technical requirements include:

• Plexiglass barriers to separate interviewers and interviewees
• Signs and markings on the floors (see Fig. 2)
• Video and audio technology to monitor the test users from outside the room and, where necessary, to give them further instructions
• Disinfection stand
• Waste bins that can be opened by foot

Step 3: prepare the standard operating procedures and the safety strategy

The safety strategy includes a standard operating procedure that describes all the phases of the usability test:

a) Selecting the test users

The manufacturer or usability lab it has selected should ensure that the test users are representative and do not belong to a risk group. The Robert Koch Institute has published information on which people are in risk groups.

Manufacturers should exclude these population groups where possible. If this is not possible because this population is made of up of the representative users, manufacturers must explain this to these people and have them sign a separate consent form.

This information and consent form must also address the specifics regarding the use of FFP2 masks. Usually, their regular use actually requires a health check.

You can find a checklist with recruitment criteria below.

b) Information for test users

The test users should be given a leaflet in advance with information on the usability test process and the safety measures they have to observe.

You can find further information on leaflets and checklists below.

c) Planning usability tests in the time of corona

The usability lab should plan the tests so that as few people as possible are present and come into contact at the usability lab.

This will generally mean that the laboratory facilities are utilized less than normal.

d) Welcoming of the test users

The usability lab employees must

• Welcome the test users while maintaining a safe distance
• Offer them the disinfectant dispenser and, if necessary, a mask
• Give the test users information about the facilities and route layout
• Escort the test users to the waiting room

Of course, the employees themselves must wear a mask.

e) Instruction and training of the test users

The laboratory staff should also be aware of issues other than how to actually conduct usability tests in the time of corona. For example, the staff should also take the test users nervousness into account and remind them that the aim is not to achieve desired results.

However, in times of corona, this communication is made more difficult by wearing masks.

Nevertheless, during this instruction and any training sessions required, the employees and test users must respect the rules regarding social distancing.

Therefore, the laboratory should determine in advance (and for each device specifically) who should stand where and when. We recommend marking these positions on the floor and documenting them in the “summative evaluation plan”.

f) Carrying out usability tests in the time of corona

The standard operating procedure and the safety strategy should also describe how the usability tests are to be carried out and should, therefore, regulate the following:

• Monitoring of test users
  Test users should only be monitored from a safe distance or via video cameras. This will require a high number of video cameras so that the test users can be observed from all angles.
  This does not just relate the test persons’ interaction with the device (e.g., what they do with their hands), but also their facial expressions and emotional reactions. The cameras must enable you to film the corresponding close-ups.
• Moderator interventions
  It may be the case that moderators have to intervene in the event, e.g. to reset a device. Moderators should be encouraged to maintain a safe distance from the test users even in these situations. They should ask the test users to step away from the device and, if necessary, leave it on a table.

Additional information on how usability lab staff should behave can be put in a leaflet.

Step 4: prepare information sheets and checklists for usability testing in the time of corona

a) Information sheet for usability lab employees

The information sheet should contain all instructions that are not specific to a particular usability test. These include, for example, the following requirements:

General rules

• Anyone who comes into contact with test users must wear a mask.
• Used masks must be disposed of in lockable waste containers.
• The general and usability test-specific standard operating procedures, including the recruitment checklist, must be followed.

Cleaning

• The waste containers must be emptied daily.
• The rest rooms must be cleaned daily.
• Daily checks must be carried out to ensure that the disinfectant, soap and paper towel dispensers have been filled.
• In the event of contamination with feces, blood or vomit, after they have been removed, the dispensers must be prophylactically disinfected by wiping them with a disposable cloth soaked in disinfectant. Rubber gloves must be worn for this. Diaper changing pads must be disinfected immediately after use.
• The test materials and the equipment used (pens, tablets, laptops) and furnishings at the premises (door handles, light switches, fittings in the rest rooms, table surfaces) must be cleaned and disinfected after every session.

Dealing with test users / test participants, conducting usability tests

• The test persons must be given masks.
• Test persons must be offered disinfectants to disinfect their hands before the start of the usability tests.
• A minimum distance of 1.50 meters must always be kept between the people involved.
• Before and after the sessions, the test rooms must be ventilated for a few minutes. If possible, the windows should be left open during the sessions.
• The waiting rooms must be ventilated several times a day.
• Incentives must be packed in envelopes before the sessions and then placed on a test person's table to ensure there is no physical contact between employees and test persons.

b) Checklist with recruitment criteria

The checklists for the recruitment of test users should take the following into account:

1. The test user is actually a representative user
2. The test user does not belong to the risk group according to the Robert Koch Institute

For this second point, the Johner Institute uses a short checklist during recruitment.

Recruitment criteria_COVID19Download

c) Information sheet for test users

The information sheet for test users should ensure that the test users

1. Know the safety strategy and therefore
   1. Know exactly what they're getting into
   2. Don’t worry unnecessarily
2. Comply with the safety rules and
3. Can check for themselves whether they meet the conditions for participation

The Johner Institute's information sheet contains the following sections:

Background information on COVID-19

• Information on the transmission paths: approx. 50% droplet infection, 40% infection by aerosols, 10% smear/contact infections
• Information on risk groups
• References to relevant publications

Requirements for participation

• Who is in a risk group?
• What signs and symptoms the test users should look out for?

Information on their own behavior

• Bring a mask
• Follow the social distancing rules
• Do not touch the laboratory staff, e.g., no shaking hands
• Avoid small unnecessary touches of objects with the hands (e.g., door handles, elevator buttons)
• Wash and disinfect hands
• Coughing and sneezing etiquette

Information on the usability laboratory's safety strategy

This section describes the usability laboratory’s structure and the safety concept as described in this article.

Step 5: carry out the usability test

Once all these preparations have been made, all that remains is to follow the plans. If the usability lab team notices any problems, they should improve the specifications immediately.

These specification documents are part of the QM system

The in-house data protection officer should be involved in the creation and revision of these specifications because, on the one hand, the names and contact information have to be taken so that chains of infection can be traced but, on the other, data protection requirements still have to be complied with.

Conclusion, summary

Necessary initial efforts...

Usability tests in the times of corona place minimum requirements on facilities and technical equipment. They require more precise planning.

This means an initial outlay in terms of time and cost. However, this effort will pay off because we are going to have to live with the virus for months, if not years. Manufacturers cannot get by without usability tests in the meantime.

…that will pay off

Once the safety strategy is in place, and once the plans, the information sheets and instructions are available and are being followed, there is nothing else standing in the way of conducting legally compliant usability tests even in the time of corona.

In any case, the regulatory requirements are no reason to delay summative evaluations and, as a result, the authorization and marketing of medical devices.

Contact us right away if you want to test the usability of your medical device and fulfill the requirements that make the quick marketing of your usable devices possible.

Author: Immanuel Bader