Tuesday, May 5, 2020
If manufacturers don’t conduct a legally compliant performance evaluation of their in vitro diagnostic medical device (IVD), they aren’t just running the risk of problems during the authorization process.
They are risking patient safety. Therefore, the IVDR sets strict requirements for this performance evaluation.
Find out how the requirements of the IVDR for the performance evaluation differ from those of the IVDD to make sure you achieve conformity as quickly as possible.
The most common mistakes can be avoided in eight steps.
In this section, you will learn about the potential negative consequences of a poor performance evaluation. As an IVD manufacturer, you should make absolutely sure to avoid them.
The purpose of in vitro diagnostic medical devices (IVDs) is to provide information from human samples, such as blood and tissue, that allows conclusions to be drawn about, for example, physiological or pathological processes in the body.
An IVD is used to identify, for example:
This information is crucial for making a diagnosis and, therefore, for the further treatment of a patient. The IVD performance evaluation must ensure that the information provided is correct and accurate, and that it provides the intended clinical benefit.
However, the information generated using an IVD can be incorrect in several respects:
There are numerous reasons for such errors. Examples are:
There is a risk that incorrect information could have far-reaching consequences for patients.
False positive results generally trigger an unnecessary cascade of measures that harm the patient, or at least cause them unnecessary stress:
False negative results, in contrast can lead to critical delays to urgently needed therapies or even have an impact on public health.
The performance evaluation doesn’t just enable IVD manufacturers to ensure that the risks for patients have been minimized, it also enables them to ensure that the consequences for their own company have been minimized as well.
In this section, you will be introduced to important concepts and definitions that you will need to know in order to understand the legal texts.
Fortunately, the IVDR contains several definitions:
Definition: Performance evaluation
“Assessment and analysis of data to establish or verify the scientific validity, the analytical and, where applicable, the clinical performance of a device”
IVDR Article 2(44)
This definition uses other defined terms:
Definition: Scientific validity of an analyte
“Association of an analyte with a clinical condition or a physiological state”
IVDR Article 2(38)
The basis for each IVD is the analyte that is used to identify a physiological state or disease. For example, there is a link between the coronavirus (SARS-CoV-2) and the disease Covid-19. Similarly, there is a link between the level of prostate-specific antigen (PSA) in the blood and the risk of a man suffering from prostate cancer.
Definition: Analytical performance
“Means the ability of a device to correctly detect or measure a particular analyte”
IVDR Article 2(40)
To ensure that the analyte can also be reliably measured using the method chosen, IVD manufacturers should check the analytical performance of their device. To continue with the examples: The analytical performance of an IVD is its ability to identify the coronavirus as accurately as possible (i.e., without false-positive or false-negative results) or to determine the PSA concentration as accurately as possible.
Definition: Performance of a device
“Ability of a device to achieve its intended purpose as claimed by the manufacturer. It consists of the analytical and, where applicable, the clinical performance supporting that intended purpose”
IVDR Article 2(39)
This definition uses another term, “clinical performance,” that we need to understand.
Definition: Clinical performance
“Ability of a device to yield results that are correlated with a particular clinical condition or a physiological or pathological process or state in accordance with the target population and intended user”
IVDR Article 2(41)
IVD manufacturers must provide proof of an IVD’s clinical benefit based on data on scientific validity, analytical performance and clinical performance. Scientific validity refers to the association between the analyte and the disease or physiological state. After all, for the IVD to deliver clinical performance, it must have a reliable and precise method for measuring the analyte. Such a method demonstrates the ability of the IVD to determine clinical, physiological and pathological states or identify processes, and is therefore a vital aspect of the device's performance.
It is also important to note that clinical performance does not refer to an absolute ability but, in fact, depends on the intended purpose and thus the “target population” and “intended user.”
The clinical performance of an IVD may be good for “normal” patients but not for patients undergoing chemotherapy because the accuracy of its measurement is affected by cytostatics.
A device's performance may be excellent for professional users, but not for laypersons. Therefore, the IVDR stresses that an IVD’s performance depends on its intended purpose.
The clinical evidence, i.e., the proof that a device is safe and achieves the intended clinical benefit, is provided by the performance evaluation. This performance evaluation assesses clinical performance, analytical performance, and scientific validity.
This section will give you an overview of
You will also find out what deltas there are between the IVDD and IVDR that you may need to close.
Both the IVD Directive 98/79/EC (IVDD) and the IVD Regulation 2017/746 (IVDR) establish, in their respective Annex Is, that the general safety and performance requirements must be complied with. Demonstrating this compliance is part of the performance evaluation.
The IVDR states that in order for a device to be placed on the market or put into service:
“A device shall meet the general safety and performance requirements set out in Annex I which apply to it, taking into account its intended purpose. [...] Demonstration of conformity with the general safety and performance requirements shall include a performance evaluation.”
Article 5, IVDR
Article 56 requires manufacturers to confirm that the general safety and performance requirements according to Annex I have been met. Manufacturers also have to carry out a performance evaluation to provide this confirmation.
This performance evaluation includes (as outlined above) the demonstration of the scientific validity of the analyte, the analytical performance and the clinical performance.
The focus is on demonstrating the following in particular:
The article refers to the requirements established in Annex XIII, Part A with regard to the performance evaluation.
This annex specifies how IVD manufacturers must plan, carry out and document the performance evaluation. Manufacturers should also refer to ISO 20916:2019, which the amended IVDR of May 03, 2019 directly references.
“The rules on performance studies should be in line with well-established international guidance in this field, such as the international standard ISO 20916 on clinical performance studies using specimens from human subjects, currently under development, so as to make it easier for the results of performance studies ...”
IVDR, Recital 66
ISO 20916:2019 was published in May 2019. Its title is “In vitro diagnostic medical devices – Clinical performance studies using specimens from human subjects – Good study practice.”
ISO 20916:2019 describes in detail the requirements for the planning and performance of clinical performance studies. It supplements the IVDR with regard to the description of the organization, the roles involved and the requirements for conducting such a study.
There are overlaps with the IVDR, particular with regard to the content of the clinical performance study plan (according to IVDR, Annex XIII, paragraph 2.3.2). ISO 20916 calls this plan the “clinical performance study protocol (CPSP).”
Annexes A to F of ISO 20916:2019 are dedicated to the specific requirements for interventional and other types of performance studies in accordance with Article 57 of the IVDR. The IVDR itself provides detailed specifications for these studies in Annex XIV.
Annex II describes the technical documentation requirements. In paragraph 6 of this annex, the IVDR describes what the performance evaluation, verification and validation should contain.
Annex VII describes the requirements to be met by notified bodies. Understanding what notified bodies have to look for during conformity assessment activities helps us to work out what is important in the performance evaluation and where the interfaces are.
According to Annex VII, paragraph 4.5.4 of the IVDR, the focus of the assessment should be on a manufacturer’s procedures and the documentation regarding:
The annex explicitly requires notified bodies to take into consideration the available common specifications, guidance and best practice documents.
They must also assess the results of the literature searches, of the verifications and validations performed, and of other tests. Notified bodies must also assess the packaging, stability studies, and results of the shelf life tests.
The assessment of the performance evaluation must expressly cover:
It is noteworthy that the notified body must ensure:
This gives us hope that there will be significantly more transparency and comparability in future when it comes to information on an IVD’s performance.
The requirements of the MPG (German Medical Devices Act) refer to the In Vitro Diagnostics Directive (IVDD). Here too the focus is on the intended purpose. And here too, IVDs must – taking into account their intended purpose – meet the relevant essential requirements according to Annex I. (see MPG § 6, paragraph 2).
“Evidence of the suitability of in vitro diagnostic medical devices for the specified in-tended purpose should be provided through performance evaluation based on appropriate data. The performance evaluation should be based on:
– data from scientific literature which cover the intended use of the medical device and the techniques involved in its use as well as a written report containing a critical evaluation of these data or
– the results of all performance evaluation studies or other appropriate tests.”
MPG, Section 19, paragraph 2
In Annex I, paragraph 3 the IVDD lists the essential performance parameters that an IVD must meet (unless the parameters can be excluded with good reason). Furthermore, in Annex I, paragraphs 1 and 2, the directive describes the requirements for the safety of the device and the acceptability of the benefit-risk ratio.
In the third paragraph of Annex III, indent 11 requires “adequate performance evaluation data showing the performances claimed by the manufacturer and supported by a reference measurement system.”
The standard EN 13612:2002 is harmonized under the IVDD. The standard, which is entitled “Performance evaluation of in vitro diagnostic medical devices” specifies “the responsibilities and general requirements for the planning, conduct, assessment and documentation of a performance evaluation study.”
Further information of conducting and documenting performance evaluations can be found in the guidance documents from the IMDRF(formerly the GHTF). They represent the current state of the art under the IVDD and were the basis for the procedure as it is now described in the IVDR.
In some places you can find word-for-word the same requirements. Therefore, the requirements of the IVDR are – contrary to what a lot of manufacturers have claimed – not new at all, rather they have been the state of the art since 2012.
Manufacturers should pay particular attention to the following documents:
In this section, we will explain step-by-step how you can conduct target-oriented legally compliant performance evaluations.
First write a standard operating procedure for IVD performance evaluations and integrate it into your QM system. How to create a standard operating procedure is described below.
Make sure that the standard operating procedure covers the entire IVD performance evaluation process. It should cover the complete life cycle of the device and goes far beyond development.
Note: Although ISO 13485 does not explicitly require a process for the performance evaluation to be defined, it does require, in section 4.2.1 e), all other documentation specified by applicable regulatory requirements. And this requirement is found in the IVDR in Article 56 and Annex XIII, among others. The IVDR describes the performance evaluation as a continuous process. It emphasizes the close interface with risk management (Annex VII, Section 4.5.4 and Annex XIII section 1.1).
The next step is to write a specific performance evaluation plan for each IVD. Make sure that you include all the content required by Annex XIII, paragraph 1.1 of the IVDR in the performance evaluation plan.
In the performance evaluation plan, you should describe the current state of the art, with regard to the medically or diagnostically relevant field as well as in terms of the technology.
Research in guidelines, the scientific literature and standards, and document this literature search clearly.
Now development can start. Demonstrate the scientific validity of the analyte(s) to be detected or measured by the IVD as early as possible in the development cycle.
Carry out a further systematic literature search. Document this search clearly, e.g., the search strategy, the sources used and the criteria for selecting the data. Statements by professional societies and expert opinions can also be included.
If this evidence is not sufficient, carry out feasibility studies. Their results can be used to prove the scientific validity of the analyte, particularly in the case of novel analytes.
Summarize all the results in a scientific validity report. Add the report to the performance evaluation file, which is itself part of the technical documentation.
Note: Use your literature search report in the continuous process to update the performance evaluation and during the post-market performance follow-up (PMPF).
Now you can start with the analytical performance evaluation as part of the IVD verification process. Describe your planned procedure and the concrete experimental design in an analytical performance evaluation plan (verification plan).
If you have a complex IVD system (e.g., consisting of an IVD device, IVD assay and IVD software), plan the verification of
As well as the analysis itself, you should also plan the sampling and sample handling precisely because reproducible and accurate analysis results can only be obtained if the conditions during the sampling and preparation of the samples are controlled.
Then conduct the evaluation according to the plan and finally summarize the results of the analytical performance studies in an analytical performance report. Please also evaluate the results in this report.
Next move on to the clinical performance evaluation, which will show the diagnostic accuracy of your IVD.
If you use (supplementary) literature data, document your systematic literature search clearly.
Write another plan for the clinical performance studies – the clinical performance study protocol (CPSP). Make sure that you include the aspects required by Annex XIII, paragraph 2 of the IVDR as well as those required by ISO 20916:2019. You should also make sure that the plan provides for the collection of data for all parameters listed in Annex I, paragraph 9.1 b).
Finally, summarize the results of the clinical performance evaluation of your IVD in a clinical performance report.
Depending on the type of device you have, you now need to demonstrate its stability. This evidence is particularly important for IVD assays and in vitro diagnostic reagents.
First plan the stability studies. For this, follow the advice in the EP25 guidance document from the Clinical and Laboratory Standards Institute (CLSI). When preparing your plan, make sure that you take all stability aspects mentioned by the IVDR in Annex II, paragraph 6.3 into account. These are in particular:
Lastly, summarize all the results on scientific validity, on analytical performance and clinical performance in the performance evaluation report.
In the report, you should evaluate the clinical evidence in the light of the current state of the art in medicine and demonstrate your device's positive benefit-risk ratio.
Create a plan for the post-market performance follow-up plan (PMPF plan). You should take into account the requirements of Annex XIII, Part B, paragraph 5.2 when doing so.
Note: PMPF is the acronym of the English term “post-market performance follow-up”.
Conduct this follow-up for your CE-marked and marketed device and summarize the results in the PMPF report. Use the results of this report to update the performance evaluation report.
Note: The performance evaluation is a continuous process. The performance evaluation activities are carried out during the device's entire life cycle. Therefore, manufacturers are obliged to continuously surveil their device and update the performance evaluation with new information and data that becomes available.
This section will describe the biggest mistakes you should avoid. Use this list as a checklist to avoid unnecessary hassle and costs.
Manufacturers often formulate the intended use in a more general and less specific way:
As a result, the performance also has to be demonstrated with no limitations on, e.g., indications and target populations. The scope of the studies required grows exponentially. Furthermore, requirements that apply, for example, for performance studies in children, pregnant or breastfeeding women (see Articles 61 and 62) are easily overlooked.
If manufacturers do not systematically research the state of the art and do not clearly document this research, there will be a lack of
Lack of knowledge of the state of the art leads to manufacturers not setting device requirements and, for example, not specifying sufficient controls for the device (e.g., self-tests).
If the literature review is not conducted in full and documented clearly, the manufacturer cannot demonstrate that they have met the performance requirements. Authorities and notified bodies will notice this and refuse authorization.
The typical mistakes include missing or incomplete performance evaluation plans. This leads, for example, to analytical performance studies without a statistically valid experimental design.
Reasons for inadequate plans include a lack of knowledge of the state of the art with regard to planning, which is described in the CLSI standards. Incidentally, the IVDD harmonized standard EN ISO 18113-1:2011 “In vitro diagnostic medical devices – Information supplied by the manufacturer (labelling) – Part 1: Terms, definitions and general requirements” refers to these CLSI standards.
The following mistakes are often observed during clinical performance studies:
Manufacturers often hope to get by without clinical performance data from studies. This is possible in principle. However, it requires equivalence between the IVD to be evaluated and the comparator device. This technical and medical or diagnostic equivalence must be analyzed in detail and the evidence documented. Unfortunately, the information required is often missing.
It is not enough to just demonstrate analytical performance. Manufacturers must also prove the scientific validity of the analyte. This is forgotten because the IVDD does not explicitly require this, even though it is the state of the art according to GHTF/SG5/N7:2012.
The IVD experts at the Johner Institute can provide support to IVD manufacturers with all performance evaluation activities and phases:
The Johner Institute’s IVD team can
Get in touch if you want to conduct a quick and legally compliant performance evaluation that will be waved through by your notified body with no problems.
The performance evaluation is as important for IVD manufacturers as the clinical evaluation is for medical device manufacturers. Both are based on data that are already available in the literature or that have to be collected through studies.
The nature of the studies and the data differ considerably. When evaluating the performance of IVDs, manufacturers must demonstrate:
Each of these aspects requires its own plans, data and evaluations.
The performance evaluation requirements established in the IVDR and the standard ISO 20916:2019 it references are significantly more wide-ranging than those established by the IVDD. However, under the IVDD, the GHTF and CLSI documents and EN 13612:2002 were already considered the state of the art. So, there aren’t any significant differences.
This means that IVD manufacturers who already comply with the relevant regulations are well prepared for the IVDR. However, a lot of IVDs will be subject to a conformity assessment by a notified body for the first time under the IVDR. All IVD manufacturers should prepare well for the detailed review of the performance evaluation file.
The situation is challenging for manufacturers who must comply with the IVDR’s requirements for the performance evaluation and the MDR's requirements for the clinical evaluation because their devices are subject to both regulations. However, this can be avoided with a clever regulatory strategy.
Do you have any questions about this article? Contact us via our micro-consulting team. The Johner Institute’s IVD team will be happy to help.