We developed the AI Assurance by Design Framework from our experience of complex engineering programmes and high-stakes AI systems.

Model performance answers only part of the trust question. Accuracy, calibration and robustness matter, but they do not establish whether the complete system merits confidence in its intended use. Confidence may instead be weakened by poor workflows, unclear interfaces, supplier dependencies, weak handovers between teams or evidence that no longer represents the deployed system.

When assurance starts too late, teams may have completed the required documents without answering the question that matters: does the available evidence justify using this system under its current operating conditions?

We use assurability to mean designing a system so that a credible assurance argument can be developed, tested and maintained over time. Assurance is the argument and evidence supporting a decision; assurability is what makes that argument possible. The framework puts this into practice through six principles:

1. Start with clear claims

“The model is 94% accurate” reports a result. On its own, it is neither adequate assurance evidence nor an assurance claim. It does not define the task, dataset, threshold or operating conditions behind the figure. Nor does it state what the system is expected to do, where that expectation applies or what happens when conditions change.

A useful claim ties performance to a defined decision and context, with clear limits, fallbacks and monitoring. These claims focus the assurance effort on evidence that supports what matters, rather than isolated measures that are easy to produce but difficult to relate to operational use.

Illustrative scenario 1: Sepsis detection in healthcare

Consider a hospital using an AI-supported system to identify patients at risk of sepsis and prompt clinical review.

The relevant operational claim concerns whether the complete system supports timely and appropriate escalation for deteriorating patients. Model accuracy is evidence for part of that claim, not a substitute for it.

That claim depends on more than model performance:

• observations must be linked to the correct patient;
• alerts must reach someone able to act in time;
• false alarms must not make meaningful alerts easier to overlook; and
• clinicians must be able to interpret and challenge the recommendation.

Our assurance scope must therefore include the route from patient data to clinical action, including the points at which information may be delayed, misdirected, misunderstood or ignored.

2. Design the system to be assurable

Assurance by Design means making early choices that allow the system’s behaviour, limits and dependencies to be understood, tested and controlled. These can include explicit operating limits, controlled assumptions, traceability, degraded modes and defined points at which evidence must be refreshed. By doing this work upfront, you make it much easier to produce evidence later on.

Assurance effort should be proportionate to the decision and its consequences. More demanding claims require stronger evidence where failure could cause serious harm, decisions are difficult to reverse or recovery options are limited. The aim needs to be sufficient relevant, credible and current evidence for the decision being made, not the largest possible evidence pack

Illustrative scenario 2: Coastal surveillance and threat triage

Consider an AI-enabled system that fuses radar, video and Automatic Identification System data to detect and prioritise small craft near a naval base. It alerts operators and recommends escalation options, but does not authorise engagement.

Detection performance addresses only part of the operational claim. Early analysis might also identify:

• benign traffic being escalated as a threat;
• operators accepting recommendations without adequate challenge during periods of high workload;
• degraded or conflicting sensor inputs producing apparently confident outputs; and
• changes in traffic or vessel behaviour invalidating the evidence used at release.

Those findings should influence the design before operating arrangements become fixed. The programme might narrow the permitted operating conditions, give the watchkeeper the information and authority needed to challenge recommendations, define bounded behaviour for degraded sensor conditions, and manage supplier components as dependencies whose changes could affect the assurance case.

Assurance has practical value when it changes how the system is built or operated. If it begins only after the important decisions have been settled, it risks becoming a retrospective attempt to justify them.

3. Progress on evidence, not paperwork

Assurance should inform decisions throughout the lifecycle rather than act as a final check. At inception, teams should define the claims and identify where they could fail. During design, they should examine whether the requirements, controls and architecture support those claims. At release, they should decide whether the evidence is adequate for the proposed use. In operation, they should monitor whether the conditions supporting the assurance case still hold.

Completing an assurance activity is not the same as producing adequate evidence. Documents, process records and test reports show what work was done. Their value depends on whether the work was relevant to the operational claims and conducted under representative conditions. Assurance must test the realism and limits of that evidence, identify residual risks and uncertainties, and ensure that they are meaningfully addressed.

Illustrative scenario 3: Collision avoidance in space

Consider an autonomous collision-avoidance function on a satellite in low Earth orbit. It estimates collision risk and may recommend or execute a manoeuvre using tracking data that are uncertain, delayed or incomplete.

A strong collision-risk model supports the assurance argument, but cannot establish the safety of the deployed system on its own. A manoeuvre that is safe when calculated may be unsafe by the time it is executed. Tracking sources that appear independent may share an underlying error, while two spacecraft may manoeuvre in response to the same alert. Avoiding one conjunction may also create another or consume limited propellant unnecessarily.

The assurance case must address situations in which the system cannot reach a reliable answer. For example, it should define how conflicting or potentially compromised inputs are handled, when authority transfers to an operator and what the system does if communication is unavailable. In this context, neither stopping automatically nor continuing on unreliable evidence is necessarily safe. Robust work to mitigate real-world operational consequence like this example are often skipped, but are critical towards the assurance of the system as a whole.

4. Recognise common threats to assurance

The same weaknesses recur across otherwise different AI-enabled systems. Evidence may measure what is convenient rather than what matters operationally. Behaviours that are not visible in component testing may appear when software, operators and processes interact. Operating limits may be poorly defined or difficult to enforce, while human oversight that appears credible on paper may fail under real workload and time pressure. Even strong evidence can lose relevance as data, software, suppliers and operating conditions change.

These threats are easy to miss when reviews examine documents or components separately. What appears to be an evidence gap may instead reveal a weakness in how the system was designed, integrated or operated. More testing or another entry in the risk register will not fix an underlying design problem.

5. Address assurance threats through design first

The strongest response to an assurance threat is usually to reduce it through design. For example, where false positives could lead to inappropriate action, the wider system might check the output against other evidence, restrict the actions it can trigger or require another decision before action is taken. These controls reduce the effect of false positives at source, rather than depending on a user to identify and correct every one.

Where the threat cannot be removed through design, technical controls should limit its likelihood or consequences. Monitoring should identify changes in performance, assumptions or operating conditions, while governance should assign the authority and responsibility to respond. This gives a preferred order of response:

• remove or reduce the threat through design;
• limit the likelihood or consequences through technical controls;
• monitor for changes that could weaken the controls or invalidate the evidence; and
• use governance, procedures and training to manage the remaining risk.

These layers must work together, but controls built into the system are generally less dependent on consistent human action. Training, reviews and management processes should reinforce the design rather than compensate for weaknesses that could reasonably have been removed earlier.

6. Assure the system, not just the model

Model testing provides evidence about the model under defined test conditions. System assurance must judge whether that evidence remains relevant when the model is integrated, deployed and used to support real decisions.

The effect of an AI model depends on how its output interacts with people, processes and other technology. Those interactions can introduce failure modes that are absent from the model itself. An inaccurate model output may cause harm. So may an accurate output that is misinterpreted, linked to the wrong record or presented through a workflow that encourages excessive reliance. Changes to data, software, infrastructure or supplier components may also weaken or invalidate the evidence supporting the original claims.

The assurance scope should therefore include the people, components and dependencies that can materially affect the operational claim. The assurance argument should trace how a model input becomes a decision or action, and where that process could fail or change.

Now put assurance into practice

The framework helps teams test their assurance argument while the system can still be changed, rather than discover fundamental problems at release. It is not a checklist and cannot replace professional judgement. Effective assurance requires people to interpret evidence, challenge assumptions and make decisions under uncertainty for the system at hand. Applied early, the framework can expose weak claims, missing evidence and design problems before they become expensive to correct. It also makes the effect of later regulatory or market changes easier to assess.

Synoptix helps organisations apply the framework proportionately, whether they need an internal assurance approach or an assurance strategy for a complex, high-stakes system. For a specific or time-sensitive question, our rapid assurance reviews provide an independent assessment of the evidence, assumptions and remaining uncertainty.

Download the AI Assurance by Design Framework to apply these principles to your own systems and assurance decisions.