Trend analysis is a legal obligation of all medical device manufacturers, especially in "post-market surveillance."
Manufacturers must not fail in the selection and application of suitable statistical methods for trend analysis. This is because the focus of authorities and notified bodies is increasingly shifting to monitoring post-market activities.
This article provides a quick introduction to this complex topic, outlines the legal obligations, and offers seven concrete tips for implementation. It also lists the seven most common mistakes that manufacturers should avoid at all costs.
The following roles need to help identify, assess, and respond to trends in the post-market phase:
Trend analysis is relevant for medical device manufacturers in post-market surveillance (PMS) to
These roles must
In addition, manufacturers must specify these activities in procedure and work instructions and document the outputs of these activities, e.g., in reports.
This article is not about the analysis of market trends and the prediction of market developments. Rather, it describes the application of statistical methods to evaluate time series of post-market data and to identify emerging (problematic) developments.
Many regulatory documents have requirements for trend analysis (see Tab. 1).
ISO 13485 Chapter
ISO 14971 and ISO 24971
Determine which data is collected and how often
Annex 3 Part 1.1
21 CFR 803.17
Systematically collect data according to this plan
Article 88 (83)
21 CFR 803.17; IMDRF
Article 88 (83)
Determine and justify methods for statistical evaluation and trend analysis
Article 88 (83)
10.3 (ISO 24971)
21 CFR 803, IMDRF
Statistically evaluate frequency of incidents according to plan, identify trends
Article 88 (83)
10.3 (ISO 24971)
21 CFR 803.53,
Set threshold values
Annex III, Part 1.1
Check the acceptability of the risks
Article 88 (83)
21 CFR 803.10
React when threshold values are exceeded (corrective action, reporting to the authorities)
Article 88 (83)
8.2.2, 8.2.3, 8.3.3, 8.5.2
21 CFR 803
Identify causes of problems
Among others 8.5.2
21 CFR 803
Tab. 1: Regulatory requirements for trend analysis
Manufacturers shall report (…) any statistically significant increase in the frequency or severity of incidents that are not serious-incidents or that are expected undesirable side-effects that
The MDR and IVDR thus oblige manufacturers to find out whether the assumptions in the risk management file are still correct. If this is no longer the case, they might have to report even non-serious incidents (Fig. 1).
The requirements of the IVDR in Article 83 are largely identical. However, it adds:
"... or of any significant increase in expected erroneous results established in comparison to the stated performance of the device [...]."
MDR and IVDR require manufacturers to plan for post-market surveillance.
b) The post-market surveillance plan shall cover at least:
– suitable indicators and threshold values that shall be used in the continuous reassessment of the benefit-risk analysis and of the risk management as referred to in Section 3 of Annex I
These two requirements result in the following:
1. Manufacturers must constantly check whether their risk management assumptions are still correct on the basis of the collected PMS data.
2. They must set threshold values in accordance with the risk policy and check whether these are exceeded and, if necessary, take corrective and preventive measures.
3. If these values are exceeded to the point where the benefit-risk ratio is no longer acceptable, they must report this identified trend to the authority.
This reporting requirement is independent of the vigilance cases that manufacturers must always report.
Thus, several decision steps are necessary (see Fig. 1):
1. is the "increase in severity of non-serious incidents or expected undesirableside-effects" significant?
2. is the impact on the benefit-risk ratio significant?
3. is the risk-benefit ratio still met?
Which methods are helpful in making the decision and whether changes are significant is described in this article below.
MDR and IVDR use the term "trend" but do not define it. Therefore, it is helpful to use other definitions of the term.
In contrast, EU regulations define when an increase is "significant:"
The significant increase shall be established in comparison to the foreseeable frequency or severity of such incidents in respect of the device, or category or group of devices, in question during a specific period as specified in the technical documentation and product information.
Article 88 MDR
This, in turn, means that one should compare the actual values for probabilites and severities of risks with the originally assumed values in the risk analysis. How large this difference may be and when measures are taken must be specified by the manufacturers via the threshold values. There are several methods to specify these thresholds. They are described below.
All activities described below must be described by the manufacturers in work and standard operating procedures.
EU regulations, ISO 13485, ISO 14971, ISO 24971, and ISO/TR 20416 provide more specific guidance to manufacturers. Manufacturers should not only collect and evaluate customer feedback; other data sources include:
Manufacturers should determine not only the data sources but also the attributes they want to capture per record. For example, customer complaints can be documented using the following attributes:
Set up time series
Unique device identification (e.g., via UDI)
Determine product type and other dependent attributes
Type of hazard
Decide on reporting obligation
Probability and severity of possible harm
Decide on reporting obligation
Place of use of the device
Find out the cause of the problem (e.g., use environment)
Production line in which the device was manufactured
Find out the cause of the problem (e.g., production)
Person who has identified the problem
Finding out how well "quality gates" work
Tab. 2: Examples of attributes for capturing data
Which attributes the manufacturer records depends on the objective. This should be described in the respective standard operating procedure, e.g., in the complaint management process.
Manufacturers are required to specify the frequency of data collection. This depends on the type of device, the technologies used, and the questions to be answered (see Tab. 3).
Data to be collected
Feedback initiated by customers must always be immediately evaluated in order to identify serious incidents. The statistical evaluation of processed further feedback can then be carried out on a periodical basis, e.g., quarterly. The frequency of customer surveys depends on the objective.
NIST Cybersecurity Database
This data should be collected and pre-evaluated continuously and thus automatically. If this is not (yet) possible, a weekly frequency is recommended.
(Note: UL 2900-2-1 requires that patches are rolled out after two weeks).
Clinical literature on the state of the art
Tab. 3: Examples of the frequency of data collection
The frequency of data collection also determines how quickly trends can be identified.
The laws already determine the minimum set of questions for which the manufacturers must evaluate the data.
For these questions, manufacturers must also monitor trends and decide if there is a statistically significant increase.
In order to be able to detect statically significant changes, manufacturers must use appropriate methods (plural!).
Based on the diversity of the medical devices in the market it is not meaningful to define a single trending procedure valid for all devices.
Many methods are available for trend analysis:
The choice of method(s) always depends on the research question.
In the pharmaceutical environment, such trend analyses have been common practice for decades. Accordingly, one will find what one is looking for in the literature there, such as in the book Static Methods of Trend Analysis.
Here is an example from the "Nelson Rules": The fifth rule states that a trend exists if there are two or even three points in a series that are more than two standard deviations away from the mean in one direction (see Fig. 2).
Benefit from the support of the Johner Institute's post-market team to create or review your PMS plan and all the methods it specifies specifically for your devices. So, you can pass the next audit with ease.
Get in touch here.
According to the MDR/IVDR, significant increases in the frequency or severities of potential harms must be reported if they "have a significant impact on the benefit-risk-determination and could lead to unacceptable risks" (see also Fig. 1).
A change is considered significant if it causes severities or probabilities to fall into a class other than that assumed (all arrows in Fig. 3).
Such a change is reportable if the associated risks move into the unacceptable range (see black-filled arrows in Fig. 3).
In this approach, defining the thresholds, as shown above, is helpful: A trend is significant if the probability of potential harms (or associated severities) falls into a class higher than the assumed class.
This is easier than using complex statistical procedures to identify trends and set thresholds.
If enough data is available, averages (e.g., monthly) can help smooth out short-term fluctuations.
Check with the PMS experts at the Johner Institute to see how they can review your PMS plans and give you specific advice for improvement. They will also help you evaluate your data.
Contact us right away via the web form with a suggested appointment for a free, non-binding consultation.
When products are new to the market, or there are only a few instances of a device on the market, there is no data basis for many statistical procedures. Trends can hardly be derived from individual data points.
Then, depending on the risk, manufacturers have to make an increased effort to collect data:
MDR and IVDR require a proactive approach. It is not in compliance with the law, especially for devices with high risks, to simply wait until sufficient data is available for trend analysis.
ISO 20416 also allows for qualitative consideration. The Johner Institute has included this in PMS plans for many manufacturers, which has been accepted by notified bodies.
If devices are discontinued, evaluations based on sales figures are no longer useful. It is generally better to standardize the values on the basis of usage.
If no information is available on this, the question arises as to whether the manufacturer is adequately fulfilling its obligation for post-market surveillance.
The following seven tips will help manufacturers with statistic evaluation and trend analysis.
Don't overcomplicate anything. It is better to start with only one method for evaluation than not to start at all. For example, you can largely meet the regulatory requirements with the procedure described in the second part of chapter 4 c).
Once you have established this evaluation procedure, you can add more data, methods, and evaluations to learn more about your devices (and users).
Many manufacturers feel overwhelmed by the amount of data. If you don't know which data to collect and evaluate with which priority, think "from behind," i.e., starting with the patients.
To do this, think in terms of a fault tree analysis of which events/hazards could endanger the safety of the patients. In this way, you get an idea which data give an indication of these events/hazards. Start with this data.
Also, look at the clinical evaluation recommendations: here, there should be an indication of what data are essential to capture in post-market surveillance or post-market clinical follow-up.
The IMDRF has developed numerous codes that you can use to code problems, causes, etc., across the organization. These codes are required in reports (e.g., the PSUR). Therefore, use these codes from the beginning of your data collection and processing.
Don't think of collecting and analyzing data as a chore required by law. Trend analyses help you learn more about your company, your devices, and your customers and support necessary decisions.
You'll get relevant answers to questions like:
In order to answer these questions, you need to collect the necessary data (e.g., about suppliers, materials, and production facilities).
You should not start to think about this data only when the devices are on the market. Rather, an early and continuous evaluation is helpful and also required by ISO 13485, for example, in the context of process validation and monitoring.
The intervals at which you collect and evaluate data should be variable.
1. The intervals should be dependent on the type and significance of the data (see Tab. 3).
2. The intervals should depend on the amount of data. As the amount of data increases, it is possible to perform evaluations and gain insights at shorter intervals.
3. The intervals should be in line with the risk that you may be able to capture or assess through this data.
All of this assumes that you automate as many activities as possible:
Outsource diligence work to Johner Institute's post-market services. This allows you to benefit from economies of scale and focus on dealing with identified trends. You will achieve more for your company with less effort.
The obligations of manufacturers through MDR and IVDR apply to all devices. Manufacturers cannot invoke the transition periods. These have long since expired with regard to post-market surveillance.
Limiting data collection and analysis to customer complaints does not meet regulatory requirements:
Even if manufacturers analyze multiple data sources, they may still miss trends if they do not collect and evaluate the necessary attributes.
The analysis of the worldwide data does not reveal any problems with electrical safety. However, if the manufacturer had analyzed the data for different countries separately, he would have noticed that the number of broken insulations has been increasing in the USA for months.
Post-market surveillance and data evaluation are processes that must interact closely with other processes (see Tab. 4).
Risk management serves as input for post-market surveillance (e.g., acceptance criteria). Conversely, PMS serves as input for risk management.
Post-market surveillance provides the information to decide on and execute trend reporting in particular.
When determining what data is collected, the "design" of the device must be considered. For example, all SOUP defined in the development must be monitored. Conversely, static evaluation of post-market data provides valuable information on how to improve the design of the devices.
Manufacturers must specify the critical attributes of the supplied devices, materials and services. Whether these are fulfilled and sufficient to ensure the quality of the devices can be seen, among other things, in the evaluation of post-market data.
Tab. 4: Examples of the interaction between the evaluation of post-market data and other processes
More than one year of evaluation periods are usually not accepted by authorities and notified bodies. But even periods of up to one year can be too long for statistical evaluation and trend analysis: trends are detected too late (only after one year) or not at all (because they average out over the year).
Tip 6 (see above) helps to define suitable intervals.
Auditors easily embarrass manufacturers with the following questions:
Without computer support, the data can neither be collected nor evaluated in a meaningful way. Therefore, Excel, self-written scripts, ticket systems, and statistical programs are used.
All these are computerized systems (or parts of them), which are subject to the obligations of ISO 13485 according to computerized systems validation. Do you have these validation documents?
Johner Institute is ISO-13485 certified and can provide validated post-market services and demonstrate the appropriate regulatory requirements.
The manufacturers make other mistakes when selecting and applying the methods:
Many manufacturers are not aware that they must report not only serious incidents but in certain cases also "statistically significant increases in the frequency or severity of non-serious incidents”.
Statistical evaluation and trend analysis, however, are more complex than most manufacturers realize. They are required by law and are under increasing review.
Therefore, manufacturers should dig deeper into the subject and build the necessary competencies.
Support from the Johner Institute
Avoid regulatory hassles, problems with unsafe devices, and unnecessary expenses. What helps: Outsource your post-market activities, such as post-market surveillance and regulatory monitoring.
Contact us right away with a proposed appointment for a free and non-binding meeting.
During this meeting, you can also clarify with the PMS experts of the Johner Institute how they can review your PMS plans and give you concrete advice for improvement. They will also support you in evaluating your data.