Microsoft Warns: AI Agents May Pose Internal Risks Due to Unreliable Frontier Models

On Monday, Microsoft unveiled new security measures aimed at safeguarding businesses against the increasing risks posed by AI agents. This announcement accompanies ongoing warnings from Microsoft regarding the potential threats associated with AI technology.

A key feature introduced is Guardian Agents, which is designed to oversee the AI agents utilized by organizations. This functionality is part of Project Perception, initially revealed in June.

Project Perception was conceived as a response to advanced hacking models, including Anthropic’s Mythos and Fable, as well as OpenAI’s Daybreak, which focuses on identifying security vulnerabilities. When first launched, Project Perception included three types of agents: a “red team” that simulated hackers, a “blue team” aimed at detecting the red team’s activities and evaluating security weaknesses, and a “green team” responsible for remediation.

The initiative was predicated on the belief that AI threats would stem from external sources. However, Microsoft’s CEO, Satya Nadella, has recently cautioned that the real danger might lie with the AI agents that businesses have implemented, likening them to potential rogue employees or “insider threats.”

Earlier this month, in a private communication, Nadella emphasized the necessity of reevaluating trust toward AI model providers, despite their claims of security and reliability. He advocates for organizations to adopt separate oversight systems, independent from AI developers, to monitor and regulate the agents these systems deploy.

“We are deploying complex systems with access to our most confidential information and granting them the authority to perform critical actions on our behalf,” he stated. “We cannot simply pass on the responsibility for what intelligence does for us. A provider’s assurances do not absolve us of that duty. … We need to disentangle the supply of intelligence from the authority that governs it.”

While announcing Guardian Agents, Hayete Gallot, head of Microsoft’s security division, was particularly candid.

“The issue is that frontier models, irrespective of their origin, cannot be deemed trustworthy,” she mentioned, clarifying that this is not necessarily due to them being malicious or easily compromised, but because of their unpredictable nature. They are programmed to complete tasks but may not adhere to “organizational norms or judgment.”

Additionally, these agents often have extensive access to sensitive data and critical tools, which raises significant security concerns.

Microsoft’s approach is both unexpected and strategically advantageous. The unexpected aspect is that Microsoft has close ties with both OpenAI and Anthropic through investments. Given that OpenAI is also a major client of Microsoft’s cloud services and Microsoft operates its own AI lab, the company is subtly undermining some of its own business relationships.

The strategic element, however, is that Microsoft seeks to position itself as the provider of security solutions for agents, encouraging businesses to choose its offerings over those from Anthropic and OpenAI. Despite potential conflicts of interest, the rationale is compelling; monitoring software for agents should be distinct from the entities that develop them, similar to how audits are conducted independent of the organizations being examined.

As Nadella articulates: “We must separate the supply of intelligence from the authority over it.”

Microsoft is capitalizing on this increasing demand in various ways. In addition to Guardian Agents, it has integrated an agent detection capability into its Microsoft Defender program, which identifies up to 60 types of agents employed by employees, including Claude Code, Codex, and OpenClaw. Moreover, it has introduced functionalities that restrict agents’ access to information and systems and adjust their permissions dynamically based on their tasks. This includes needing human authorization for agents that appear to be making potentially dangerous decisions.

The perception of agents as unreliable insiders is resonating throughout the technology industry.

Microsoft is among the founding members of Nvidia’s new Open Agent Safety Platform, which has garnered the backing of over 100 tech companies. This platform focuses on isolating agents in secure settings to monitor and intervene in their behavior, functioning similarly to a kill switch. This follows Microsoft’s launch of the Surface Laptop Ultra PCs, designed specifically for agents and powered by Nvidia chips, which incorporate inherent security and control features.