Heuristic evaluation is a process which usability experts use to assess the usability of products by means of heuristics (explained in more detail below).
This article shows how you can use heuristic evaluations to meet the regulatory requirements of usability engineering very quickly and economically.
It also reveals why you should not restrict heuristic evaluation to the medical device or product itself.
A heuristic evaluation is a process which, as the name suggests, is based on heuristics. Heuristics are not synonymous with principles. A heuristic process is a rough rule of thumb, by means of which it is possible to reach a conclusion quickly and relatively accurately. In contrast, a principle is a ground rule that always applies.
Fig. 1: Heuristics versus principles. Heuristic evaluation not only uses heuristics but often uses principles and other style guides as well (see Table 1)
ISO 9241-110 mentions the seven design principles:
In usability engineering a distinction is made between heuristics and principles, and design rules and conventions.
Example of description
1. Design principles
A ground rule that always applies.
The system shows the necessary information for the user in every situation (self-descriptiveness).
Standards (e.g. ISO 9241-110)
A rough rule of thumb, which often (but not always) applies, and serves to achieve (or verify) the implementation of a design principle.
The system should have a help function.
This heuristic process supports the “learnability” principle. This applies as a matter of principle, but is not always applicable. For instance, a ticket machine does not require a help function.
Standards and subject literature
3. Design rules
A concrete rule that must, however, be able to be implemented in different ways.
Input fields for mandatory data must be able to be differentiated from input fields for optional data.
Standards and subject literature
4. Established conventions
Established standard specifications with no room for interpretation
Manufacturers’ style guides
(also standards to some extent)
Table 1: Types of style guides (from the book by Geis and Johner “Usability Engineering als Erfolgsfaktor – Effizient IEC 62366- und FDA-konform dokumentieren”* [Usability Engineering as a Success Factor – Documenting in compliance with IEC 62366 and the FDA])
Original list of heuristic processes
Jacob Nielsen developed a set of well-known heuristics back in the 1990s. These heuristics include, for example, the following:
See the full list of Norman Nielsen’s heuristics:
If we study these heuristics in more detail, we can see that they are made up of a combination of heuristics and design principles. ISO 9241-110 calls the latter “dialogue principles”:
The ISO 9241 family of standards provides a very comprehensive collection of style guides (heuristics, design rules). For example, ISO 9241-125 explains how colours are to be used for coding, how cursors are to be designed, how and where labels are to be positioned and much, much more.
Read more about ISO 9241, as well as the specifications of ISO 9241-125.
Fig. 2: ISO 9241-125 mentions heuristics and rules for designing user interfaces (click to expand)
Operating instructions can also be tested with the help of heuristics and other criteria:
If more comprehensive operating instruction tests are possible, they are performed based on the criteria of IEC 82079-1 and AAMI TIR 49.
Heuristic evaluation is above all offered in the case of formative, i.e. development-accompanying, evaluation of user interfaces. It enables relevant user problems to be identified quickly and economically.
However, heuristic analysis is not suitable for proving the usability of user interfaces. To do this, it requires participatory observations, i.e. from usability testing.
Do not just evaluate the products with heuristic evaluation but also the operating instructions and other labelling elements.
Sometimes the procedure can also be applied for products that are already on the market. For example, a company that takes over a manufacturing company and its products would be able to quickly acquire an initial appraisal of their usability.
Before the courts or in the event of disputes with the FDA, experts use heuristics to make statements about the usability of products.
A heuristic evaluation regularly involves the following steps:
As with every process, heuristic analysis has its pros and cons.
TR IEC 62366-2 explicitly mentions heuristic analysis as one of the processes of formative evaluation. It describes this process in annex E11. The technical report even mentions heuristic evaluation in the context of assessing risk control measures.
However, the description of heuristic analysis is only four sentences long. “Only” these points are worth mentioning:
In the guidance document on human factors and usability engineering the FDA also goes into heuristic analysis:
The description of the process is also just a few sentences long in this guidance document.
The concept of heuristic evaluation can often be a grey area, including in Wikipedia. Many sources not only understand it to be a test of usability by experts using heuristics. Rather, they apply all design rules and principles as test criteria.
That heuristic evaluation should be used above all as a process in formative evaluation is true. However, as mentioned above, there are also cases that go beyond this.
Heuristic analysis depends crucially on the competence and availability of experts. These experts should also originate from same environment as the future users in terms of language and culture. A German expert can only carry out provisional tests on operating instructions for the US market and vice versa.
If these requirements are met, manufacturers have a powerful and extremely economical, time-saving method on hand to be able to detect usability problems early and react to them.
In the article by Theresa Neil you can find great examples of how Nielsen’s heuristics are implemented in software applications.
t3n magazine has written an article about rules, patterns and dark patterns (No english version available) in usability engineering.
Worth a read is Zhang’s article “Using usability heuristics to evaluate patient safety of medical devices”.
In their usability labs in Germany and the USA, the Johner Institute also supports manufacturers with heuristic evaluations and IEC 62366-1 and FDA-compliant documentation.