The operational adaptations described in the previous post rarely remain temporary. Over time, they harden into culture. Culture itself does not introduce new risks though; it stabilises and hides existing ones.
AI systems are rarely used exactly as they are described in policy documents or training materials. Their real impact emerges through culture. Over time, people reinterpret what AI is “for”, how much authority it holds and how far its outputs can be trusted. These cultural dynamics shape behaviour far more than formal guidance, yet they remain largely invisible to assurance and governance regimes. This highlights the ‘above the waterline, below the waterline’ concept, where only a small proportion of assurance regimes are visible.
Understanding this shift is essential if organisations want AI to guide decisions, not make them.
From decision support to decision authority
Systems introduced as decision support are quickly reframed in practice. A recommendation becomes a default. A probability score becomes a judgement. A model output becomes something to justify decisions rather than inform them.
This happens through both anthropomorphising and instrumentalising AI. Some users attribute intent or intelligence to the system, treating it as a quasi-expert. Others see it as an efficient shortcut, a way to move faster or reduce personal accountability. In both cases, the system’s stated role erodes, replaced by an informal understanding of what it is allowed to decide, fundamental misuse of the system. Are metrics of disuse and misuse measurable behaviours for AI adoption and in turn assurabilty of AI-enabled systems?
These shifts are subtle, cumulative and rarely acknowledged.
The rise of invisible practice
As AI embeds into daily workflows, informal practices emerge. Users learn how to interpret outputs, when to ignore them and how to present them to others. These practices dominate real-world use, yet they are almost entirely absent from assurance artefacts.
Assurance frameworks typically assume prescribed usage. They rarely capture how people adapt under pressure, how outputs are selectively trusted or how exceptions become normalised. The result is a growing gap between documented use and lived use.
For organisations relying on AI to improve speed, insight or consistency, this gap matters. It is where risk accumulates, not because the technology is failing, but because the system of people and technology has quietly changed shape.
The power of organisational narratives
Cultural dynamics are reinforced by the stories organisations tell themselves. Narratives such as “AI makes us faster”, “AI makes us smarter” or “AI makes us safer” are powerful. They help justify investment and drive adoption. But they also suppress scepticism.
When AI is framed as a strategic enabler, questioning its outputs can feel like resistance to progress. Challenge becomes socially costly. Over time, this discourages critical thinking and narrows the space for raising concerns about edge cases, degraded modes or unintended consequences.
The organisation becomes invested in the success story, not the scrutiny of the tool.
When AI fades into the background
Perhaps the most significant shift occurs when AI becomes routine. Once a system is embedded, it stops being noticed. It is no longer something to be evaluated, but something that is simply there.
At this point, AI is no longer treated as a tool that requires ongoing scrutiny. It becomes part of organisational habit. Assumptions harden. Practices stabilise. And the opportunity to question whether the system still aligns with operational reality diminishes.
For AI-enabled systems, this is a dangerous moment. Risk has not disappeared. It has just become hidden behind operating procedures.
Reintroducing scrutiny without slowing delivery
Addressing cultural dynamics does not mean rejecting AI or slowing innovation. It means designing governance and assurance that recognise how people actually work.
The table below, highlights what more resilient defence organisations can do to improve the AI Assurance activities with defence evidence and key takeaways briefly identified.
| What do they do? | What difference evidence shows | Key takeaway |
| Reintroducing scrutiny without slowing delivery | Defence programmes repeatedly demonstrate that scrutiny and pace are not opposites. Programmes that integrate assurance into everyday delivery, rather than treating it as a decision gate, identify risk earlier and reduce rework later. UK defence AI assurance thinking explicitly recognises that risk arises from how systems are used in context, not just how they are designed, requiring governance that reflects real operational behaviour rather than idealised process descriptions [REF]. | Scrutiny that mirrors real work accelerates delivery by preventing late‑stage failure. |
| Observe how AI is used in practice, not just how it is described | Across MOD digital and C2 programmes, formal documentation frequently diverges from operational reality. Architectural and AS‑IS/TO‑BE studies in RAF Digital programmes show that capabilities described as advisory are often repurposed under operational pressure, with risk only becoming visible through observation of real workflows and user behaviour. AI assurance literature reinforces that testing and documentation alone cannot surface these risks; assurance must include evidence from operational use and human‑system interaction. See our case study on Synoptix’s work with RAF Digital. | If assurance stops at documentation, it misses where most defence AI risk actually emerges. |
| Create space for challenge without framing it as obstruction | Defence systems engineering and safety practice treat independent challenge as a primary risk control, not a delivery impediment. Case evidence from RAF Digital shows that structured challenge workshops, walkthroughs and early stakeholder engagement surfaced integration and operational risks before capability insertion, reducing downstream delay and cost rather than slowing delivery. | In defence contexts, challenge accelerates delivery by preventing silent or hidden failure modes. |
| Treat cultural signals as risk indicators, not soft issues | Defence and high‑stakes domains have long recognised that behavioural cues language, habits and informal workarounds are early indicators of system risk. AI assurance thinking now explicitly links these cultural signals to over‑reliance and automation bias, arguing they should be treated as evidence within assurance cases, not dismissed as training or communications issues. Systems engineering practice increasingly places human behaviour within the system boundary, recognising cultural drift as a measurable contributor to operational risk. | In defence, culture is not “soft”: it is a leading indicator of technical and operational risk. |
| Revisit assumptions as AI becomes embedded, not just at deployment | MOD‑aligned lifecycle assurance practice recognises that risk evolves as systems embed into operations. Evidence from our digital twin and decision‑support programme, on behalf of BAE shows that capabilities framed as decision aids can become de facto authorities over time, requiring scheduled re‑assessment of assumptions, accountability and oversight rather than one‑off acceptance at deployment. | AI risk in defence increases after deployment, not before. Assurance must follow the lifecycle. |
At Synoptix we combine deep systems thinking, cyber security discipline, and advanced AI/digital engineering to ensure that assurance is applied early, influences design, and addresses risk at the system level—not just at the component level. Our approach moves beyond procedural compliance to interrogate assumptions, expose uncertainty, and force meaningful trade-offs before they become embedded in delivery.
Drawing on our pedigree in high‑stakes defence programmes, we integrate assurance with engineering judgement: assessing how AI systems behave in real operational contexts, how humans actually use them, and how risks propagate across data, models, infrastructure, and mission outcomes. This enables us to identify systemic vulnerabilities—such as degraded modes, data drift, adversarial exposure, and emergent behaviours—that conventional assurance frameworks often miss.
These cultural assumptions are rarely challenged until the system is stressed. This is where the final article focuses: Stress Conditions: Where AI’s Real Risks Finally Appear.
This article is part of a series on AI assurance in practice:
1. Use, misuse, abuse and disuse: why human intent breaks AI assurance models
2. When AI fails in practice: The operational reality leaders need to address
3. When AI becomes cultural habit rather than a tool
Coming Soon:
4. Stress conditions: where AI’s real risks finally appear
Topics from this blog: AI Assurance
