This article is part of a series exploring why AI assurance fails in practice; not because models break, but because human intent, culture and pressure reshape how AI is used. Each post examines a different layer of that problem, from individual behaviour to organisational dynamics, exposing the use, misuse, abuse and disuse cycle of AI-enabled systems.
Most AI assurance models are built around what a system is designed to do. Far fewer are built around what people will actually do with it. In practice, this gap is where many AI assurance activities fail. This article, and subsequent articles, explore this gap through a single defence example: an AI‑enabled ISR triage system introduced to help intelligence analysts prioritise large volumes of sensor data. The system is designed to flag imagery or signals likely to contain objects or activity of interest, allowing human analysts to focus attention where it matters most.
On paper, assurance is straightforward. The model is tested against labelled datasets, performance thresholds are defined, and human‑in‑the‑loop processes are documented. In practice, however, this system moves through a use, misuse, abuse and disuse cycle driven by human behaviour, culture and pressure, not technical failure.
Most AI assurance activities focus on what the model is designed to do. System design often forgets that we, humans, use the system that is being designed.
This article is grounded in an example, based on a triaging intelligence, surveillance and reconnaissance (ISR) example highlighted in the Resilient Defence AI Sustainable and Operationally Effective Capabilities by Design report, published by the Alan Turing Institute.
Use of AI-enabled systems
Operational reality intervenes quickly. AI systems often break in use before they break in code. Operators adapt tools to fit tempo, workload, and mission pressure. Workarounds emerge. Safeguards are bypassed. Outputs are trusted in contexts far removed from those originally envisaged. None of this is malicious. It is a rational response to stress, incentives, and organisational culture. Yet many assurance models struggle to account for it. Our upcoming article When AI Fails in Practice: The Operational Reality Leaders Need to Address, we examine how this gap shows up in live operations.
ISR Example: Initially, the ISR triage system is used as designed. Analysts treat its outputs as indicative rather than authoritative. Flags prompt review, not action. Cross‑checks with other intelligence sources remain routine. Assurance assumptions broadly hold because operational behaviour aligns with design intent. At this stage, confidence grows. The system appears reliable. It demonstrably reduces workload. From an assurance perspective, everything looks healthy. But this phase is temporary.
Misuse of AI-enabled systems
Cultural dynamics play a decisive role. Users reinterpret what AI is “for” based on how it performs in practice, not how it is described in documentation. A decision support tool becomes a decision maker. A productivity aid becomes an authority. Over time, trust migrates from process to output, especially when the system appears to perform well under normal conditions. Traditional assurance activities rarely revisit these shifts once a system is in service. Later in the series, we explore how cultural habit and stress amplify these risks: When AI Becomes Cultural Habit Rather Than a Tool.
ISR example: As operational tempo increases, behaviour begins to drift. Analysts discover that the system is usually right under normal conditions. Under pressure, its flagged outputs start to carry implicit authority. Items not flagged are reviewed later, or sometimes not at all. The triage tool quietly becomes a gatekeeper of attention, not just a support. This is not malicious misuse. It is a rational response to volume, fatigue and incentive structures. Yet assurance artefacts rarely capture this shift. The system is still described as “decision support”, even as it begins to shape decisions by omission. Trust migrates from process to output.
The abuse and disuse of AI-enabled systems
Stress conditions are where unknown unknowns surface. Under operational pressure, behaviour changes. People rely more heavily on automation, reduce cross-checking, and narrow their focus. AI systems that appear robust in testing can behave very differently when data quality degrades, inputs are incomplete, or time is constrained. Assurance that does not explicitly explore stressed human-machine interaction is blind to some of the most credible failure modes, this is amplified further when these high stress interactions are absolutely required as part of the systems' function(s). Read more in our upcoming article: Stress Conditions: Where AI’s Real Risks Finally Appear.
Incentives amplify the problem. Delivery success is often rewarded, while friction is penalised. Teams are encouraged to adopt AI quickly, demonstrate value, and scale use. In a Defence context, the operational need on the frontline (“get working kit into the hands of war fighters”) is forcing MOD teams to ask for pace at all costs from their suppliers, incentivising them to adopt quickly, but does this come with it’s own challenge of not understanding the AI aspects innately? Most probably. Will it continue until there is a gold standard? Most likely.
ISR example: Under sustained pressure or degraded conditions, misuse hardens into abuse. During high‑tempo operations, analysts rely heavily on the AI to cope with scale. The absence of a flag is treated as evidence of absence. Edge cases that fall outside training distributions are systematically missed. At this point, the system is no longer being used within its assured bounds. It is being over‑relied upon in precisely the conditions where its limitations matter most. Importantly, the AI has not changed. Human behaviour has.
Eventually, trust fractures. A high‑profile miss, a false negative under stress, or a perceived slowdown causes some teams to bypass the system altogether. Analysts revert to manual scanning or alternative heuristics. Others continue to rely on it unquestioningly. The organisation now exhibits fragmented practice, with no shared understanding of when or how the AI should be used. Assurance documentation still assumes uniform, compliant use. Reality has diverged completely.
What is viable assurance?
Assurance activities that raise uncomfortable questions about misuse, abuse or disuse can be perceived as obstructive. Over time, this creates a bias towards assuring the AI that was intended, not the AI that is actually being used.
AI systems rarely fail because people do exactly what designers expect. They fail because people do what makes sense to them in context. If assurance does not account for that, it is assuring a fiction.
AI systems rarely fail because people do exactly what designers expect. They fail because people do what makes sense to them in context. If assurance does not account for that, it is assuring a fiction. Synoptix brings a strong systems thinking heritage to this challenge, built on 15 years supporting some of the UK’s most complex Defence programmes. By assessing how AI performs in real operational conditions and how people actually use it, we surface vulnerabilities that are often missed, including degraded performance and unexpected emergent behaviours, to build a clear, defensible case for trust in operation.
This article is part of a series on AI assurance in practice:
1. Use, misuse, abuse and disuse: why human intent breaks AI assurance models
Coming Soon:
2. When AI Fails in Practice: The Operational Reality Leaders Need to Address
3. When AI Becomes Cultural Habit Rather Than a Tool
4. Stress Conditions: Where AI’s Real Risks Finally Appear
Topics from this blog: AI Assurance
