In the previous article, we introduce how AI-enabled systems go through a use misuse, disuse and abuse cycle. This post examines how AI-enabled systems are used, in practice, and the challenges associated with this.
AI rarely fails in the way many organisations expect. It does not usually collapse because the code is broken or the model stops performing. Instead, it breaks in use. One could draw similarities with the Maccabi Tel Aviv football game, where, as a result of a hallucination generated through the use of CoPilot in which ‘false information that was generated by AI’, formed a decision to ban Maccabi Tel Aviv fans [REF]. The gap between how AI is designed to be used and how it is actually used in deployed operations is where risk quietly builds.
This is particularly true for the intelligence, surveillance and reconnaissance (ISR) AI systems highlighted in our previous article. These tools are adopted for good reasons: speed, cost. ease of use and rapid access to advanced capability. However, once deployed, they are often repurposed into environments they were never trained, tested or constrained for. The model may still be “working” according to its original metrics, but the operational context has changed.
Traditional AI assurance focuses heavily on technical performance. Accuracy, bias, robustness and validation against test data dominate assessments. While these are essential, they only tell part of the story.
In real operational settings, people adapt. Operators develop workarounds to meet time pressure. They over‑rely on outputs when systems appear confident, or under‑trust them when results clash with intuition. Tasks are subtly reallocated between humans and machines, often without formal redesign, governance or approval.
These adaptations are not failures of discipline; they are normal human responses to operational reality. But they are rarely captured in assurance artefacts, which tend to assume stable roles, predictable workflows and consistent usage patterns.
Many of the most serious AI failures do not originate from obscure technical edge cases. They emerge from day‑to‑day improvisation; human use of an AI model.
A model used slightly beyond its intended scope. A dashboard output treated as definitive rather than indicative. A decision-maker skipping contextual checks because the system has “always been right before”. Over time, these small shifts create new edge cases that were never tested, because they were never imagined in the original design. Edge cases are often emergent and interaction-driven rather than purely technical.
This is why AI can appear compliant on paper while becoming fragile in practice.
We discuss how human autonomy teaming (HAT) can be broken down into a framework, considering aspects such as instability, uncertainty or lack of robustness in the interface, within our paper published at the Annual Systems Engineering Conference (ASEC) 2026 here.
When AI enters an organisation, it does not simply automate a task. It changes how work is done. A nuance to this, is that rather than a single workflow being conducted, numerous interacting decision loops are deployed to provide a resulting decision. This distinction is key for differentiating assurance activities of AI-enabled systems from non-AI-enabled systems.
Human–AI teaming reshapes task boundaries, decision authority and accountability. People stop doing some checks because the system does them. They focus attention elsewhere. They adjust behaviour based on what the AI surfaces and what it hides. An ISR recommendation tool quickly becomes an authoritative decision point, turning a potential target into a confirmed target, with minimal oversight from a human.
These shifts invalidate many of the assumptions embedded in original assurance documentation. Yet assurance processes are rarely revisited as operational practices evolve.
To manage AI risk effectively, organisations need to complement technical assurance with operational insight.
This means:
At Synoptix, we see this challenge across sectors where complex decisions depend on integrated data, analytics and human judgement. Insight does not come from models alone, but from understanding how data, tools and people interact over time.
AI does not fail in isolation. It fails when operational reality drifts beyond what was designed, tested and assured. Recognising that reality is the first step towards building AI systems that remain trustworthy, resilient and valuable in practice.
Over time, these behavioural adaptations solidify into cultural norms, something we explore in the next article When AI Becomes Cultural Habit Rather Than a Tool.
This article is part of a series on AI assurance in practice: