How to define the state of the art in SaMD clinical evaluation?

Defining the state of the art for SaMD clinical evaluation under the EU MDR

Every SaMD clinical evaluation under the MDR compares the device with something. That reference point is the state of the art, the current body of knowledge about the clinical condition, the standard of care and the alternatives available to patients and clinicians.

In SaMD clinical evaluation the state of the art is often the weakest part of the clinical evaluation report. Teams tend to describe the technology rather than the medical context, search one database with a handful of keywords, and compare their model only with other AI products, which leaves the notified body without a clear basis for judging whether the benefits outweigh the risks.

Scarlet, the notified body specialised in software and AI as a medical device, has listed several of these pitfalls in its own guidance for manufacturers, and they match what reviewers raise most often in clinical evaluation assessments.

This article explains why the state of the art matters in SaMD clinical evaluation, what it should cover, how to run and document the literature search behind it, and how to use it to set acceptance criteria that carry through to post market clinical follow up.

Why the state of the art matters for SaMD clinical evaluation

Section 1 of Annex I of Regulation (EU) 2017/745 requires devices to achieve their intended performance and to be safe, taking into account the generally acknowledged state of the art. Section 2 requires risks to be reduced as far as possible without adversely affecting the benefit risk ratio, which can only be judged against what current practice already achieves.

Annex XIV Part A requires the clinical evaluation plan to identify and appraise the relevant clinical data, and the guidance behind it expects the evaluation to place the device within current medical knowledge. In practice the state of the art answers three questions for the notified body, namely what the patient would receive without the device, how well those options perform, and what level of safety and performance a new device must reach to be acceptable.

For software this comparison is easy to get wrong. The relevant alternative to an AI triage tool is usually the existing triage process, with its known sensitivity, delays and workload, not another algorithm. Our article on clinical data requirements for CE marking explains how the state of the art fits into the overall evidence strategy.

How MDCG 2020-1 frames SaMD clinical evaluation

MDCG 2020-1, available on the European Commission MDCG guidance page, sets out clinical evaluation for medical device software in three parts. The valid clinical association shows that the output of the software corresponds to the targeted clinical condition. Technical performance shows that the software produces that output accurately, reliably and precisely from the input data. Clinical performance shows that the output is clinically relevant in accordance with the intended purpose.

The state of the art supports all three. Literature and professional guidelines establish the valid clinical association, published performance of existing methods provides the benchmark for technical performance, and outcomes achieved under current practice define what clinically relevant performance means for the target population.

What the state of the art should cover

A complete description of the state of the art for SaMD usually addresses the following areas.

Clinical condition and patient population

The condition, its epidemiology, natural course and burden, and the population defined in the intended purpose. This section establishes why the clinical problem matters and which outcomes are relevant to patients.

Current standard of care and alternatives

The diagnostic, monitoring or therapeutic options currently used for the condition, whether they involve software or not, as described in clinical guidelines from professional societies and in the literature. For many SaMD products the main alternatives are clinical assessment by a professional, established scoring systems, laboratory tests or imaging read by a specialist.

Benchmark devices and performance levels

Comparable devices and methods, including other software where it exists, together with the performance they achieve in metrics such as sensitivity, specificity, agreement with reference standards or time to decision. These figures become the basis for acceptance criteria in the clinical evaluation plan.

Known risks and limitations

The hazards associated with current practice and with comparable devices, including failure modes typical of software and AI, such as performance loss on populations or acquisition conditions not represented in training data. These feed directly into the risk management file under ISO 14971 and the quality management system.

MEDDEV 2.7/1 revision 4 was written under the directives, but its appendix A5 on the literature search and review protocol remains the reference that notified bodies use under the MDR. It expects the search to serve at least two objectives, establishing the state of the art and identifying data on the device under evaluation or an equivalent device.

MDCG 2020-13, the template notified bodies use for the clinical evaluation assessment report, asks the reviewer to examine the search documentation, including the databases used, the search terms, the selection and exclusion criteria and the appraisal of the retrieved literature. It also expects more than one database to be searched to reduce bias. Gaps in this documentation are a frequent source of non conformities.

  • Write a search protocol before searching, stating the objectives, the research questions and the methods.
  • Structure the questions with the PICO approach, describing population, intervention, comparison and outcome for both the state of the art and the device.
  • Search several sources, typically PubMed together with Embase and the Cochrane Library, and add clinical trial registries, guidelines and relevant standards.
  • Record each search string, the date it was run, the number of hits and the reasons for every exclusion, so that the search can be reproduced.
  • Appraise the included publications for methodological quality and relevance to the device, and keep full text copies of the papers used.
  • Summarise the selection process in a flow diagram and report favourable and unfavourable findings alike.

Where teams use language models to screen abstracts or draft summaries, the controls described in our article on LLMs in SaMD regulatory documentation apply, and every inclusion and exclusion decision should still be verified by a qualified reviewer.

Common pitfalls in the SaMD state of the art

  • Describing the device and its technology instead of the medical context in which it is used.
  • Comparing the device only with other AI products and leaving out the clinical methods it is intended to complement or replace.
  • Running a narrow search in a single database, with search terms chosen to find supportive results.
  • Relying only on peer reviewed articles and ignoring clinical guidelines, consensus statements, registries and standards.
  • Citing performance figures from studies in populations or settings that differ from the intended purpose without discussing the difference.
  • Treating the state of the art as a fixed chapter instead of updating it with each revision of the clinical evaluation report.

What a SaMD clinical evaluation report contains

The state of the art is one part of a larger document. A SaMD clinical evaluation report brings together the clinical evaluation plan, the device description and intended purpose, the state of the art, the clinical data on the device itself and the analysis of benefits and risks, and it is updated throughout the life of the device.

Annex XIV of the MDR and MDCG 2020-1 define what the evaluation has to establish, while MEDDEV 2.7/1 revision 4 still provides the most widely used structure for the report. Notified bodies generally expect the following content in a SaMD clinical evaluation.

  • A clinical evaluation plan that defines the scope, the clinical benefits, the outcome parameters and the acceptance criteria.
  • A description of the device, its software versions, its intended purpose and the claims made for it.
  • The state of the art, with the literature search protocol, the search report and the appraisal of the retrieved data.
  • Evidence of valid clinical association, technical performance and clinical performance, as set out in MDCG 2020-1.
  • Data from verification and validation, usability engineering and any clinical investigation or performance study.
  • An analysis of benefits and risks, the conclusions on conformity with the general safety and performance requirements and the plan for post market clinical follow up.

The qualifications of the evaluators form part of the report as well. Notified bodies expect a SaMD clinical evaluation to be carried out by people with expertise in both the clinical field and the software concerned, and to include their declarations of interest.

For a startup, the report should be planned early, because the acceptance criteria it defines determine which validation studies have to be completed before submission. This is where our SaMD clinical evaluation services usually begin.

From state of the art to acceptance criteria and PMCF

The state of the art is only useful if it leads to explicit, measurable acceptance criteria. If current practice detects a condition with a known sensitivity in a given population, the SaMD clinical evaluation plan should state the level the device must reach, with a justification based on the benefits it offers and the risks it introduces.

Those criteria then govern the verification and validation programme, the clinical evaluation and the post market clinical follow up plan required by Annex XIV Part B. For AI devices they also provide the reference against which future model updates are tested, which links the state of the art to the change control process described in our article on updating an AI medical device after CE marking.

Because medical practice and competing technologies change, the literature search should be repeated at defined intervals, and the state of the art updated whenever the clinical evaluation report is revised. A notified body will expect to see that the benchmark used for the device is still current at each surveillance cycle.

A state of the art checklist for startups

Startups preparing a first SaMD clinical evaluation can use the following steps.

  • Start from the intended purpose and list the clinical condition, population, users and decisions the device is meant to inform.
  • Identify the current standard of care and every relevant alternative, whether software based or not, using clinical guidelines as the starting point.
  • Write the search protocol, run searches in several databases, and document every step so that a reviewer can reproduce it.
  • Extract benchmark performance and safety data, and translate them into acceptance criteria with a written justification.
  • Link the acceptance criteria to the risk analysis, the validation plan and the PMCF plan.
  • Schedule updates of the literature search and the state of the art as part of the quality management system.
Defining the state of the art for SaMD clinical evaluation under the EU MDR

Frequently asked questions

What is the state of the art in a clinical evaluation?

It is the current body of knowledge about the clinical condition, the standard of care, the available alternatives and the performance and safety they achieve. Under Annex I MDR, devices must perform and be safe taking the generally acknowledged state of the art into account.

Does SaMD need a state of the art section if no similar software exists?

Yes. The comparison is with current clinical practice, not only with other software. If no comparable software exists, the state of the art describes how the condition is diagnosed, monitored or treated today and how well those methods perform.

Which guidance applies to the literature search under the MDR?

MEDDEV 2.7/1 revision 4, appendix A5, remains the main reference for the search protocol, and MDCG 2020-13 describes how notified bodies review the search documentation. MDCG 2020-1 adds the specific framework for medical device software.

Which databases should be searched?

Notified bodies expect more than one source to reduce bias. PubMed, Embase and the Cochrane Library are common choices, supplemented by clinical trial registries, professional guidelines and relevant standards.

How often should the state of the art be updated?

Whenever the clinical evaluation report is updated, and at intervals defined in the quality management system. For higher class devices this follows the cycle of the periodic safety update report and notified body surveillance.

Build your SaMD clinical evaluation with Aura Health

Aura Health supports digital health and SaMD startups in defining the clinical state of the art, writing literature search protocols, conducting systematic reviews and preparing SaMD clinical evaluation plans and reports that follow MDCG 2020-1 and meet notified body expectations.

To discuss the clinical evidence your device will need, write to partnerships@aurahealth.ch.