The same legal requirements apply to the clinical evaluation of software as to the clinical evaluation of all medical devices. This means that as a Medical Device Software (MDSW) manufacturer, you must prepare a clinical evaluation for your product just like any other manufacturer. A performance evaluation must be carried out for software that is an in vitro diagnostic (IVD) device.
That, in turn, requires clinical data on the device itself, perhaps from a clinical investigation or clinical performance study, or on a proven equivalent product. For software, this approach often makes little sense.
However, the MDCG document MDCG 2020-1: Guidance on Clinical Evaluation (MDR)/Performance Evaluation (IVDR) of Medical Device Software shows a possible alternative path for software products. The Notified Bodies have now accepted the first clinical evaluations of software prepared by us on the basis of this document.
“Clinical evaluation means a systematic and planned process to continuously generate, collect, analyze and assess the clinical data pertaining to a device in order to verify the safety and performance, including clinical benefits, of the device when used as intended by the manufacturer”
Source: MDR Article 2
“Performance evaluation means an assessment and analysis of data to establish or verify the scientific validity, the analytical and, where applicable, the clinical performance of a device”
Source: IVDR article 2
The goals are mentioned, albeit indirectly, right here: It is about proving whether the medical device
performs as promised by the manufacturer.
a) The essential difference: The interfaces
While other medical devices have many forms of “outputs” (= interfaces to the outside world), such as electrical energy (e.g., RF surgical device), electromagnetic energy (e.g., CT) or mechanical energy (e.g., ultrasound device), the “output” of stand-alone software is limited to information.
Software thus differs from other medical devices in terms of interfaces:
Therefore, for software, clinical evaluation must consider whether the information provides value to the user, for example in diagnosis or therapy selection, or whether it provides useful information to other products.
This concept will certainly sound familiar to IVD manufacturers. Thus, the benefit of an IVD always lies in the provision of appropriate medical information (see IVDR, recital 64). Clinical evidence is provided for in vitro diagnostic software through performance evaluation.
Here, we are not referring to cases in which the software is used to control medical devices. This is because in most cases the software would be an accessory to the product in question, for which a clinical evaluation should already exist.
b) Special benefit of stand-alone software
Stand-alone software serves one of the following purposes:
The Guidance Document MDCG 2019-11 “Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR” provides guidance on distinguishing software as a medical device or IVD.
c) Clinical evaluation or performance evaluation based on verification and validation
Clinical evaluators and performance assessors often question what aspects of software products they need to evaluate in the first place. Most of the time it makes sense to focus on the algorithms, e.g.:
Sometimes, these algorithms are very mundane. The question then arises as to what clinical evaluation or performance evaluation of software should yield in terms of insights. In some cases, it may be sufficient to argue on the basis of verification results.
If, on the other hand, the algorithms are more complex, proof is needed that they deliver the promised benefit, i.e., that they are clinically valid. This is precisely the approach taken in the MDCG document MDCG 2020-1.
According to MDCG, the manufacturer must provide three pieces of evidence:
In providing evidence, the manufacturer should go through five different phases:
These three types of evidence are presented below.
a) Proof 1: Scientific validity
Source: MDCG 2020-1
“... is understood as the extent to which the MDSW’s output (e.g., concept, conclusion, calculations), based on the inputs and algorithms selected, is associated with the targeted physiological state or clinical condition. This association should be well founded or clinically accepted”
Scientific validity, then, is the evidence that the “outputs” of the product (in this case, the software) are associated with a clinical or physiological condition.
One example is the calculation of a score, which must actually be a measure of a certain risk.
To demonstrate scientific evidence, manufacturers should use the following sources:
b) Proof 2: Technical performance
“... is the demonstration of the MDSW’s ability to accurately, reliably, and precisely generate the intended output from the input data”
Verification of technical performance is thus the demonstration of the software product’s ability to accurately, reliably, and precisely produce the intended output from the input data.
Manufacturers prove technical performance in the context of verification and validation (V&V). In doing so, they should draw on the following, as appropriate:
Typically, medical device manufacturers reference these V&V results when clinically evaluating software.
IVD manufacturers finally evaluate the technical performance of the software during the analytical performance evaluation of the IVD and take into account the available V&V results.
c) Proof 3: Clinical performance
“... is the demonstration of an MDSW’s ability to yield clinically relevant output in accordance with the intended purpose”
With regard to clinical performance, the manufacturer must demonstrate clinical relevance to the intended purpose of the device. The clinical relevance of medical device software is shown to have a positive impact on
These data may come from clinical studies conducted by the manufacturer as well as from other sources. Manufacturers are also encouraged to use existing data from alternative processes and products.
For example, you should refer to the following information to assess clinical performance:
d) Summary of steps for clinical evaluation of software
The following flowchart once again shows the main steps in the preparation of the clinical evaluation:
Manufacturers must actively and continuously monitor the safety, effectiveness, and performance of the MDSW after it is placed on the market.
As with all manufacturers, information may include, but is not limited to, complaints, direct end-user feedback, or newly published research/guidelines. Of particular relevance to MDSW, however, is the collection of REAL-WORLD PERFORMANCE DATA, as this is relatively easy to obtain with software.
This allows the very quick and effective
ongoing monitoring of intended clinical performance (e.g., sensitivity, specificity).
First and foremost, manufacturers must decide for themselves whether the level of data is sufficient for each step and justify this if necessary. This involves both the quantity and the quality of the data. This evaluation can be guided by the following questions:
a) Sufficient quantity
b) Sufficient quality
c) Demonstration of compliance without clinical data
In some cases, particularly for lower risk class I and IIa medical device software, clinical data is often not adequate to demonstrate compliance. Both the MDR (Article 61, Section 10) and MDCG document 2020-1 allow an exception.
Demonstration of compliance with essential safety and performance requirements can be based solely on the results of non-clinical test methods including performance evaluation, technical testing (“bench testing”) and pre-clinical evaluation. However, this must be justified in any case. The following points must be taken into account:
Intended clinical performance and manufacturer’s performance data
NOTE: Manufacturers must still prepare a clinical evaluation that includes, at a minimum, the rationale for the exemption pathway, evidence of the nonclinical test methods, and a statement on the state of the art.
If certain performance requirements are not applicable for in vitro diagnostic devices, they can be justifiably excluded. IVD manufacturers describe their intended performance evaluation strategy in the product-specific performance evaluation plan. There they also set out the state of the art. In the performance evaluation report, the results on the clinical evidence and the benefit-risk ratio are finally summarized and evaluated.
Together with the IMDRF, the FDA has already published a very similar Guidance document “Software as a Medical Device: Clinical Evaluation” published. It requires that manufacturers verify the clinical validity of the product with the clinical evaluation of medical device software or the performance evaluation of IVD software. In doing so, they demonstrate that the intended purpose is fulfilled and that there are no unacceptable risks. Some of the definitions differ from those of the MDCG. This may lead to a lot of extra work if both countries have to be served.
The FDA provides guidance on what steps need to be taken depending on certain parameters (risk class, degree of innovation, etc.).
a) Determine class for the software
FDA uses a classification from other IMDRF documents for clinical evaluation of software:
Significance of information provided by SaMD to healthcare decision
State of healthcare
Treat or diagnose
Drive clinical management
Inform clinical management
As examples of the areas of application, the FDA cites:
b) Demonstrate evidence depending on the class
Depending on the class, manufacturers must demonstrate the following:
Only for new processes
Only for diagnostic devices of class II.iii and for new procedures
Only for diagnostic products and for new procedures
Independent review (objective and competent person outside the manufacturer)
c) Determine necessary evidence
If the manufacturer has competence and capacity, the next step is to establish the necessary evidence. In other words, the manufacturer should use the above table to decide what evidence to provide:
What lies behind these three aspects is explained above.
d) Collect data
The data can be collected as described above. Common data sources at FDA also include:
e) Evaluate data and prepare clinical evaluation of software
At the FDA, the next step is to evaluate the data and write the clinical evaluation or (for IVDs) the final performance evaluation report. For medical devices the MEDDEV 2.7/1 provides guidance on the structure of clinical evaluations, even though the MEDDEV document does not have the FDA in scope.
The Audit Guarantor includes clinical assessment videos as well as clinical assessment templates with examples and instructions for completion.
f) Conclusion on the guidance documents from IMDRF or FDA
The clinical evaluation or performance evaluation effort should be adjusted based on risk. The FDA proposes proceeding as follows in the clinical evaluation of software:
The IMDRF’s text is not easy to digest. The authors do a good job of presenting the various aspects of clinical evaluation or performance evaluation of software. They refer to many other publications by the same organization.
In the document MDCG 2020-1 on clinical evaluation or performance evaluation of software, the European Medical Device Coordination Group attempts to take account of the special requirements for software products. The authors present the various aspects of the clinical evaluation of software as a medical device or IVD quite well. The approach is much more pragmatic than the previous one.
However, some things still remain vague. Some concrete examples in the appendix help to clarify what is meant, as well as when which step of the algorithm is obligatory and when it is not. Here, a more precise demarcation of the requirements (e.g., based on the classification of the functionalities or classification of the software) would be very desirable.
It also lacks any reference to other MDCG guidance documents regarding general aspects of clinical evaluation, e.g., requirements for authors.
The FDA also offers a guidance document that takes a very similar approach. This shows much more clearly when which proof characteristics are required.
As with any clinical evaluation, the requirements of the MDCG guidance document cannot be implemented without a scientifically and medically competent team.
The Johner Institute specializes in preparing clinical evaluations for medical devices for compliance with MEDDEV 2.7/1 and MDCG 2020-1 and ensuring that they are accepted by notified bodies.
The Johner Institute’s IVD team will also be happy to assist you in developing a performance evaluation strategy for your IVD software.
Please let us know (e.g., via the Contact form) if you are interested in support from the experts of the Johner Institute for clinical evaluations or performance evaluations.