Why AI impact assessments validate delivery instead of shaping design
AI impact assessments are intended to influence decisions. In practice, they often arrive too late, ask the wrong questions and ultimately reinforce delivery pathways that are already fixed. Rather than shaping design, autonomy boundaries or operating concepts, they validate them.
This is not because impact assessment are inherently flawed. It is because of how, when and why it is applied. This isn't about just doing a better assurance process; it’s about having the right mindset, understanding and knowledge, centred around engineering judgement, to make the appropriate decisions for the appropriate measures. The table below provides two examples of patterns as a result of incorrectly conducted impact assessments.
| Callout | Pattern |
| Autonomous features are added incrementally, each framed as a minor enhancement. A full impact assessment is conducted only once the system is live. By then, autonomy boundaries are embedded in workflows and user expectations. | No single decision triggers meaningful change. |
| A system is designed as “just a decision aid” but becomes relied upon in time‑critical situations. The impact assessment occurs post‑deployment, when reclassification would require redesign and delay. The assessment documents risk, rather than implements change. | Assessment rationalises reality instead of challenging it. |
This article outlines why assurance in defence increasingly functions as governance theatre. Subsequent articles will take a deeper dive into the themes identified here. Each containing a core message: Assurance only adds value when it changes decisions early enough to matter.
Assurance, after the fact
A common failure mode is timing. Impact assessments are frequently conducted once core design choices are already locked in, architecture has been selected, autonomy boundaries have been agreed and operating concepts are assumed rather than explored.
At this stage, assessment has limited leverage. It can highlight risks, but it cannot realistically alter the fundamentals of the system. The most it can do is recommend mitigations around an existing design. Redesign, de-scoping or stopping a capability is no longer on the table. The assessment becomes retrospective justification rather than prospective influence. Our upcoming article: Why AI Impact Assessments Rarely Change the Design addresses this point directly.
Risks framed as constraints, not signals
When risks are identified, they are typically framed as constraints to be managed. Safeguards are added. Processes are updated. Oversight is increased. What is rarely considered is whether the risk is signalling that the design itself is wrong.
For AI-enabled systems, this matters. Assumptions embedded in training data, model behaviour and human–machine interaction shape outcomes just as much as technical performance. Yet these assumptions are seldom stress-tested in ways that force real trade-offs. The model is viewed in isolation. The wider operational context is treated as a configuration of the model. This ignores the risks that emerge as operator apply the model in practice.
Assurance that is procedurally strong but operationally weak
Defence does not lack assurance frameworks, it lacks the expertise to effectively use the existing assurance frameworks. JSP 936 and related structures are procedurally mature and well established. The issue is how they are used. Synoptix have analysed JSP936 to uncover the challenges teams encounter implementing this framework. This analysis is based on our ability, bespoke and tailored approaches of systems engineering and assurance within high-stakes Defence programmes. You can view this report here.
Assurance activity often prioritises demonstrable compliance over the discovery of uncertainty or failure modes. Evidence that a process has been followed is valued more than insight into how the system might behave under pressure, in degraded modes or when assumptions no longer hold.
Independence is further weakened by delivery pressure, organisational proximity and incentive alignment. Assurance teams may be formally independent, but they are rarely insulated from the momentum of programmes that are already committed to delivery. This pattern is further explored in the next article of this four part series, so keep an eye out for it.
Fragmented ownership of systemic AI risk
Systemic, cross-domain AI risks such as data drift, emergent behaviour and degraded modes rarely sit neatly within one function. They span data, engineering, operations and governance.
In many programmes, these risks are fragmented across silos. Each area manages its part, but no one owns the risk end to end. Impact assessments mirror this fragmentation, reviewing components in isolation rather than examining how risk accumulates across the whole system lifecycle. The result is assurance that is thorough in parts, but blind to interaction effects.
By example, AI risks can emerge from even the most accurate and performant models. An image analysis tool can achieve 99% accuracy in classification when detecting military vehicles in testing, but still fail to detect a threat when used in the real world, where inputs drift from it’s training examples.
This article is part of a series on AI assurance within the Defence domain:
- Assurance: why defence risks turning it into Governance theatre
- Why Impact Assessments Comfort Leaders Instead of Informing Them [coming soon]
- Why AI Impact Assessments Rarely Change the Design [coming soon]
- Why Defence Assurance Struggles to Surface Real Risk [coming soon]
Topics from this blog: AI Assurance
