Agentic AI Security: The Door Was Open, Not Broken
An AI agent does not need to escape from a sandbox when someone has accidentally left the door open.
An AI agent does not need to escape from a sandbox when someone has accidentally left the door open.
That is the uncomfortable lesson behind a new disclosure from Anthropic.
After OpenAI reported that models had escaped an isolated evaluation environment and reached Hugging Face production infrastructure, Anthropic reviewed 141,006 of its own cybersecurity evaluation runs. The company found three incidents in which Claude models reached the public internet from within, or while interacting with, a third-party test environment and then gained unauthorised access to real organisations.
This was not a fictional red-team scenario anymore.
The systems were real.
The credentials were real.
The production data was real.
And the affected organisations had not detected the activity themselves before Anthropic contacted them.
That should make every enterprise currently experimenting with Agentic AI pause for a moment.
Not because it proves that AI agents have developed their own evil master plan.
It proves something much more practical: a model can follow its instructions and still cause real damage when its environment gives it access to places it should never have been able to reach.
What actually happened
Anthropic was using capture-the-flag exercises to evaluate the cyber capabilities of different Claude models. These exercises give the model a fictional scenario in which a secret, known as a flag, is hidden somewhere inside a simulated network.
The agent’s job is to find it.
Claude was explicitly told that it was operating in a simulation without internet access. But because of a misunderstanding between Anthropic and its external evaluation partner, internet connectivity was available.
The prompt said offline.
The infrastructure said online.
And infrastructure tends to win that argument.
When the agents encountered real internet-facing systems, they treated them as part of the exercise. They used relatively conventional attack techniques, including weak passwords, exposed credentials, unauthenticated endpoints and SQL injection. Anthropic described the events as being closer to a harness and operational failure than a deliberate attempt by Claude to escape its environment.
The three cases also show how quickly the impact can escalate.
In one incident, Claude reached the infrastructure of a real company whose domain name matched the fictional company used in the evaluation. It extracted application and infrastructure credentials and gained access to a database containing several hundred rows of production data.
In another case, Claude found fictional developer instructions referencing a Python package that did not exist. It realised it could publish a malicious package under the same name so that systems following those instructions would install it.
Because the agent had real internet access, the package was published to the public Python Package Index. It remained available for around one hour and was downloaded and executed by 15 real systems. One of those systems belonged to a security company whose automated scanner installed the package, allowing the malicious code to run and expose credentials.
The third agent scanned roughly 9,000 internet-facing targets after it could not reach its intended fictional target. It eventually compromised a real application. Later, the model recognised that the system was unrelated to the evaluation and stopped the attack on its own.
Anthropic started its review and suspended the affected cyber evaluations on July 23. The company identified the three incidents the following day and published its findings on July 30.
The model is only one part of the system
The first reaction to an incident like this is often to ask whether the model was aligned correctly. That question matters. But it is not enough. An AI agent is not just a model.
It is a model connected to tools, identities, APIs, data, memory, networks, execution environments, third-party services and business processes. Together, these components form the agent harness.
And that harness is where intention becomes action. A chatbot can suggest a command. An agent can execute it on it’s own.
A chatbot can explain how a package repository works. An agent can create an account, publish a package and wait for someone to install it.
That difference changes the security model completely.
We can no longer treat Agentic AI as a clever user interface sitting on top of a language model. Once an agent can use tools and act across systems, its identity, permissions, network paths and operational boundaries become part of the AI safety architecture.
A prompt can tell an agent that it is offline. Only network controls can make it offline. A system instruction can tell an agent not to touch production. Only authentication, authorisation and explicit allowlists can prevent it from doing so. A governance guideline can require human oversight.
Only a real approval gate can stop an action before it is executed.
This is why agent security cannot rely on behavioural controls alone. It needs defence in depth, just like every other critical enterprise platform.
Why this matters beyond the AI laboratories
It would be easy to treat this as a laboratory problem.
It is not.
Enterprises are beginning to connect agents to service-management platforms, development environments, customer databases, cloud subscriptions, communication tools, ERP systems and industrial processes.
The more useful the agent becomes, the more access it usually receives.
And the more access it receives, the greater the potential blast radius.
For a general productivity agent, a containment failure might expose documents or trigger an incorrect workflow. That is already serious.
In defence environments, an agent might reach sensitive partner systems, mission-related interfaces or external infrastructure that resembles a simulated target.
In healthcare, it could interact with patient data, clinical platforms or connected medical systems under the false assumption that they belong to a test environment.
In manufacturing and industrial OT, an agent equipped with scanning, automation or terminal tools could reach production networks, engineering systems or exposed industrial endpoints.
The agent does not need malicious intent.
It only needs an objective, enough capability and an incorrect understanding of its surroundings.
That is the part enterprises must take seriously.
Governance must move from policy to execution
Many organisations still govern AI as if it were primarily a content-generation technology.
They define acceptable-use policies. They establish review boards. They document responsible AI principles. They create long presentations explaining what AI should and should not do.
That is a necessary beginning.
But an AI agent is not merely creating content. It can use an identity, access data, call tools, modify records, initiate transactions and interact with third parties.
The governance model must therefore govern action, not only output.
A governance document that never reaches the execution layer is ultimately just a PDF with ambition.
Microsoft’s Agentic AI maturity model makes the same distinction. It describes mature agent governance as a combination of enterprise-wide standards, identity and data controls, observable behaviour, clearly defined human oversight, lifecycle ownership and active cross-functional governance. It also warns against what it calls governance theatre: formal processes that create overhead without improving real security or operational outcomes.
For me, an enterprise governance model for AI agents needs five connected layers.
1. Clear ownership and accountability
Every agent needs a named owner.
Not just the developer who built the first prototype. Not a generic project mailbox. And definitely not the mysterious “business” mentioned in a responsibility matrix.
There should be a business owner accountable for the agent’s purpose and outcome, a technical owner responsible for its operation, and a risk owner who approves the boundaries within which it may act.
An autonomous action must never lead to the organisational equivalent of everyone slowly looking around the meeting room and asking:
“Whose agent was that?”
Ownership also needs an expiration date. Agents should be reviewed, renewed or retired. Otherwise, a small experiment created for one workshop can become a forgotten production component with active permissions two years later.
That is how shadow IT becomes shadow AI.
2. Risk classification based on capability
Not every agent requires the same governance model.
An agent that summarises approved documents does not create the same risk as one that modifies customer data, deploys software or operates close to production infrastructure.
A practical classification should therefore consider both impact and autonomy.
An informational agent reads and summarises approved information.
An advisory agent produces recommendations but cannot execute them.
A transactional agent can modify data or perform defined business actions.
A privileged agent can use administrative, development, security or production capabilities.
A critical agent operates in regulated, safety-relevant or industrial environments.
The higher the category, the stronger the requirements must become.
That includes approval gates, testing, isolation, monitoring, logging, identity controls, network restrictions and human intervention points.
Microsoft’s maturity guidance similarly recommends applying different governance levels according to agent purpose and criticality rather than treating every agent the same.
3. A central agent registry
You cannot govern what you cannot see.
Every enterprise needs an inventory of its agents. That inventory should show who owns each agent, which model it uses, which tools it can call, which identity it operates under, what data it can access, where it is deployed and when its approval expires.
Microsoft Agent 365 follows this idea by treating agents as first-class Microsoft Entra identities. Its registry is designed to provide an inventory of agents, ownership information and governance data, including agents built in Foundry, Copilot Studio or discovered elsewhere in the tenant.
Microsoft Foundry Control Plane extends the same principle across fleets of agents, models and tools. It provides central visibility, compliance information, security signals and cost data across projects and platforms.
The registry should not be a passive list.
It should become the control point for approvals, attestations, access reviews, policy assignment, cost ownership and retirement.
Because building agents is becoming easier.
Finding all of them six months later will be the less entertaining sequel.
4. Runtime controls that cannot be argued away
Governance must be technically enforced while the agent is running.
That means individual workload identities instead of shared service accounts. It means least-privilege permissions, short-lived credentials, approved tool catalogues and explicit network allowlists.
It also means putting additional approval gates in front of high-impact actions.
An agent may be allowed to prepare a software deployment, but not release it.
It may draft a financial transaction, but not approve it.
It may identify a security vulnerability, but not exploit an external system.
It may recommend a production change, but not execute it without confirmation.
These boundaries should not depend on whether the model remembers a sentence in its system prompt.
They must be enforced by the identity platform, network architecture, API gateway and workflow engine.
The Anthropic incident is the perfect example. The system prompt told Claude that the internet did not exist. A simple deny-by-default egress policy would have made that statement technically true.
That is the difference between a rule and a control.
5. Continuous monitoring and the ability to stop
Agent governance does not end when the agent reaches production.
Models change. Prompts change. Tools change. APIs change. Data changes. The surrounding environment changes.
An agent that behaved correctly last month may behave differently after one of those components is updated.
That means organisations need continuous monitoring of both performance and behaviour.
Microsoft Foundry provides evaluation, production monitoring and distributed tracing for AI applications and agents. Tracing can show model calls, tool invocations, agent decisions and dependencies across a multi-step execution. Its monitoring capabilities can track latency, errors, token consumption and quality signals through Azure Monitor Application Insights.
This is more than debugging.
It is evidence.
It shows what the agent did, which identity it used, which tool it called and what happened next.
For high-risk agents, organisations should also monitor DNS queries, outbound network connections, unusual credential use, package publication, file transfers and unexpected tool chains.
And there must be a kill switch.
Not a committee meeting next Thursday.
Not an emergency change request waiting for approval.
An immediate, tested and technically effective way to stop the agent, revoke its identity and block its network access.
The operating model should be federated
There are two bad extremes in AI governance.
The first is complete decentralisation. Every team builds what it wants, connects whatever systems it needs and invents its own controls. That creates agent sprawl, inconsistent security and unclear accountability.
The second is complete centralisation. Every experiment requires multiple boards, months of review and a stack of forms thick enough to stop innovation through gravity alone.
The better model is federated governance.
A central AI Council or AI Center of Excellence defines standards, platforms, risk classifications and escalation paths. Security, legal, compliance, privacy and data teams define their mandatory boundaries. Business domains remain responsible for the agents they deploy and the outcomes those agents create.
Microsoft’s maturity model similarly recommends a cross-functional AI Council for high-impact and ambiguous cases while embedding responsibility into delivery and operations.
The goal is governed autonomy.
Teams should be able to build and innovate quickly, but only inside enterprise boundaries that are clear, observable and technically enforced.
Governance also includes cost
Security is only one side of agent governance.
Agent fleets can consume models, tokens, APIs, tools, search services and compute continuously. An agent may operate overnight, repeat failed tasks, call expensive models unnecessarily or create long chains of actions without producing business value.
In other words, autonomous agents can also create autonomous cloud bills.
The governance model should therefore connect every agent to usage, cost and value.
Who pays for the agent?
What business outcome does it support?
How many tokens and tool calls does it consume?
Does its value justify its operating cost?
Could a smaller model perform the same task?
Is the agent still being used?
Microsoft Foundry Control Plane includes fleet-level views for agent activity, token consumption, resource usage and cost anomalies. That connects operational governance with FinOps and makes cost another observable part of the agent lifecycle.
An agent that is secure but economically uncontrolled is still not well governed.
What critical agent testing should require
Critical AI evaluations should begin with the assumption that prompts will eventually be misunderstood, ignored or contradicted by the technical environment.
Isolation therefore needs to be demonstrated, not assumed.
Test environments should use deny-by-default egress policies with tightly scoped allowlists for any required destinations. Agents should receive temporary identities with the minimum permissions needed for the evaluation. Test data, test domains and simulated services should be clearly separated from real systems.
Actions such as publishing packages, creating external accounts, accessing production endpoints, transferring data or running destructive commands should require independent approval.
The environment should also be monitored in real time.
Not only after the test has finished.
Anthropic itself concluded that validating internet paths, monitoring logs and reviewing transcripts more thoroughly could have prevented the incidents or revealed them sooner. The company also acknowledged that third-party evaluation environments require the same hardening and monitoring as internal infrastructure.
That is an important point.
Outsourcing the test does not outsource the risk.
Third-party tools, evaluation partners, model providers and external agent frameworks all become part of the trust boundary.
The Cloud Advisor standpoint
My conclusion is simple:
An agent does not need to break out of a sandbox when the sandbox accidentally has an open door.
Critical AI testing needs technically proven isolation, real-time egress control, individual identities, observable tool use and immediate shutdown capabilities.
Not merely a prompt telling the model that it is offline.
The Anthropic incidents do not prove that agents are uncontrollable. They prove that powerful agents must be treated as operational systems.
They need an identity.
They need an owner.
They need a defined operating area.
They need observable behaviour.
And they need someone with both the authority and the technical ability to stop them.
Agentic AI will create enormous opportunities.
It will automate complex work, accelerate decisions and connect systems in ways that were difficult to imagine only a few years ago.
But autonomy without hard boundaries is not innovation.
It is unmanaged execution at machine speed.
Stay clever. Stay governed. Stay in control.
The Cloud Advisor,
Uwe Zabel
🚀 Curious how Agentic AI, cybersecurity, governance and enterprise cloud architecture must evolve together? Follow my journey on The Cloud Advisor’s book of stories—where cloud, AI, and business strategy converge. Or ping me directly—because building the future works better as a team.