Clinical evaluation of software: Three proofs of conformity

The same legal requirements apply to the clinical evaluation of software as to the clinical evaluation of all medical devices. This means that as a Medical Device Software (MDSW) manufacturer, you must prepare a clinical evaluation for your product just like any other manufacturer. A performance evaluation must be carried out for software that is an in vitro diagnostic (IVD) device.

That, in turn, requires clinical data on the device itself, perhaps from a clinical investigation or clinical performance study, or on a proven equivalent product. For software, this approach often makes little sense.

However, the MDCG document MDCG 2020-1: Guidance on Clinical Evaluation (MDR)/Performance Evaluation (IVDR) of Medical Device Software shows a possible alternative path for software products. The Notified Bodies have now accepted the first clinical evaluations of software prepared by us on the basis of this document.

1. Quick recap: What is a clinical evaluation or a performance evaluation?

The goals are mentioned, albeit indirectly, right here: It is about proving whether the medical device

• provides the benefit promised by the manufacturer,
• is safe and does not pose any risks that have not already been identified and assessed as acceptable in the risk analysis, and

performs as promised by the manufacturer.

Software thus differs from other medical devices in terms of interfaces:

1. User-product interface, often graphical user interface (GUI)
2. Product-product interface, especially data interfaces

Therefore, for software, clinical evaluation must consider whether the information provides value to the user, for example in diagnosis or therapy selection, or whether it provides useful information to other products.

This concept will certainly sound familiar to IVD manufacturers. Thus, the benefit of an IVD always lies in the provision of appropriate medical information (see IVDR, recital 64). Clinical evidence is provided for in vitro diagnostic software through performance evaluation.

b) Special benefit of stand-alone software

Stand-alone software serves one of the following purposes:

• Diagnosis or provision of diagnosis-relevant information (medical devices/IVD)
• Selection of appropriate therapies, e.g., selection of an appropriate drug or decision on chemotherapy or radiation (medical devices/IVD)
• Performing the therapy itself, e.g., calculating the insertion angle of a surgical instrument, radiation planning, dose calculation (medical devices only)

These three types of evidence are presented below.

a) Proof 1: Scientific validity

Source: MDCG 2020-1

Scientific validity, then, is the evidence that the “outputs” of the product (in this case, the software) are associated with a clinical or physiological condition.

One example is the calculation of a score, which must actually be a measure of a certain risk.

To demonstrate scientific evidence, manufacturers should use the following sources:

• Scientific literature (peer-reviewed articles, conference papers)
• Recommendations by the authorities
• Scientific databases, including those with results from clinical studies
• Expert opinions, e.g., in books and guidelines

b) Proof 2: Technical performance

Verification of technical performance is thus the demonstration of the software product’s ability to accurately, reliably, and precisely produce the intended output from the input data.

Manufacturers prove technical performance in the context of verification and validation (V&V). In doing so, they should draw on the following, as appropriate:

• 1. Verification activities, e.g.:
  • Unit tests
  • Integration tests
  • System tests
• 2. Validation activities, e.g. based on:
  • Curated databases or registers
  • Reference databases
  • Previously collected patient data

Typically, medical device manufacturers reference these V&V results when clinically evaluating software.

IVD manufacturers finally evaluate the technical performance of the software during the analytical performance evaluation of the IVD and take into account the available V&V results.

c) Proof 3: Clinical performance

4. Clinical evaluation of software after it has been placed on the market (post-market)

Manufacturers must actively and continuously monitor the safety, effectiveness, and performance of the MDSW after it is placed on the market.

As with all manufacturers, information may include, but is not limited to, complaints, direct end-user feedback, or newly published research/guidelines. Of particular relevance to MDSW, however, is the collection of REAL-WORLD PERFORMANCE DATA, as this is relatively easy to obtain with software.

This allows the very quick and effective

• timely detection and correction of malfunctions,
• detection of systematic abuse,
• understanding of user interactions and

ongoing monitoring of intended clinical performance (e.g., sensitivity, specificity).

5. Assessment of the level of evidence and data

First and foremost, manufacturers must decide for themselves whether the level of data is sufficient for each step and justify this if necessary. This involves both the quantity and the quality of the data. This evaluation can be guided by the following questions:

a) Sufficient quantity

• Do the data support the intended use, indications, target populations, clinical claims, and contraindications?
• Have the clinical risks and analytical performance or clinical performance been investigated?
• Have relevant MDSW characteristics such as data input and output, algorithms used, or type of interconnection been considered in generating the data to support the performance of the product?
• What is the level of innovation or history in the market (how extensive is the body of scientific knowledge)?

b) Sufficient quality

• Was the type and design of the study or test appropriate to meet the endpoints?
• Was the data set appropriate and up to date (state of the art)?
• Was the statistical approach adequate to reach a valid conclusion?
• Have all ethical, legal, and regulatory considerations or requirements been taken into account?
• Is there any conflict of interest?

c) Demonstration of compliance without clinical data

In some cases, particularly for lower risk class I and IIa medical device software, clinical data is often not adequate to demonstrate compliance. Both the MDR (Article 61, Section 10) and MDCG document 2020-1 allow an exception.

Demonstration of compliance with essential safety and performance requirements can be based solely on the results of non-clinical test methods including performance evaluation, technical testing (“bench testing”) and pre-clinical evaluation. However, this must be justified in any case. The following points must be taken into account:

• Manufacturer’s risk management
• Consideration of the special characteristics of the interaction between the product and the human body

Intended clinical performance and manufacturer’s performance data

The FDA provides guidance on what steps need to be taken depending on certain parameters (risk class, degree of innovation, etc.).

a) Determine class for the software

FDA uses a classification from other IMDRF documents for clinical evaluation of software:

As examples of the areas of application, the FDA cites:

• “Treat or diagnose”
  • Treat
  • Provide therapy to a human body using other means
  • Diagnose
  • Detect
  • Screen
  • Prevent
  • Mitigate
  • Lead to an immediate or near-term action
• “Drive clinical management”
  • Aid in treatment
  • Provide enhanced support to safe and effective use of medicinal products
  • Aid in diagnosis
  • Help predict risk of a disease or condition
  • Aid to making a definitive diagnosis
  • Triage early signs of a disease or condition
  • Identify early signs of a disease or condition.
• “Inform clinical management”
  • Inform of options for treatment
  • Inform of options for diagnosis
  • Inform of options for prevention
  • Aggregate relevant clinical information
  • Will not trigger an immediate or near-term action

b) Demonstrate evidence depending on the class

Depending on the class, manufacturers must demonstrate the following:

c) Determine necessary evidence

If the manufacturer has competence and capacity, the next step is to establish the necessary evidence. In other words, the manufacturer should use the above table to decide what evidence to provide:

1. Analytical validity
2. Scientific validity
3. Clinical performance

What lies behind these three aspects is explained above.

d) Collect data

The data can be collected as described above. Common data sources at FDA also include:

• Scientific literature
• Results of the verification and validation of the software
• Existing post-market data for own and/or third-party equivalent products
• Clinical trial results

e) Evaluate data and prepare clinical evaluation of software

At the FDA, the next step is to evaluate the data and write the clinical evaluation or (for IVDs) the final performance evaluation report. For medical devices the MEDDEV 2.7/1 provides guidance on the structure of clinical evaluations, even though the MEDDEV document does not have the FDA in scope.

The IMDRF’s text is not easy to digest. The authors do a good job of presenting the various aspects of clinical evaluation or performance evaluation of software. They refer to many other publications by the same organization.

7. Summary

In the document MDCG 2020-1 on clinical evaluation or performance evaluation of software, the European Medical Device Coordination Group attempts to take account of the special requirements for software products. The authors present the various aspects of the clinical evaluation of software as a medical device or IVD quite well. The approach is much more pragmatic than the previous one.

However, some things still remain vague. Some concrete examples in the appendix help to clarify what is meant, as well as when which step of the algorithm is obligatory and when it is not. Here, a more precise demarcation of the requirements (e.g., based on the classification of the functionalities or classification of the software) would be very desirable.

It also lacks any reference to other MDCG guidance documents regarding general aspects of clinical evaluation, e.g., requirements for authors.

The FDA also offers a guidance document that takes a very similar approach. This shows much more clearly when which proof characteristics are required.

As with any clinical evaluation, the requirements of the MDCG guidance document cannot be implemented without a scientifically and medically competent team.

The Johner Institute specializes in preparing clinical evaluations for medical devices for compliance with MEDDEV 2.7/1 and MDCG 2020-1 and ensuring that they are accepted by notified bodies.

The Johner Institute’s IVD team will also be happy to assist you in developing a performance evaluation strategy for your IVD software. 

Please let us know (e.g., via the Contact form) if you are interested in support from the experts of the Johner Institute for clinical evaluations or performance evaluations.