Skip to main content
Bureau of AI
Category: Adversarial Security

Security of AI Agents

Also known as: AI Agent Security, Agentic AI Security
Simply put

Security of AI agents refers to the practice of protecting AI systems that can act on behalf of users—planning tasks, using tools, and taking actions across workflows—from being misused, manipulated, or causing harm. It covers both threats aimed at the agent itself and risks that arise from what the agent is allowed to do. In many current approaches, this involves keeping agents under human control, limiting their permissions, and constraining their actions.

Formal definition

As commonly framed in emerging literature and industry guidance, security of AI agents is the discipline of protecting agentic applications—systems capable of planning, tool invocation, memory use, and autonomous action across workflows—against both threats to the agent and risks arising from the agent's use. It extends traditional computing security properties to autonomous software acting on behalf of users, and typically emphasizes principles such as well-defined human controllers, carefully limited agent powers, and observable or constrained agent actions, alongside protection of the underlying systems, data, and tools the agent can access. This is an evolving area without a single authoritative definition; treatments vary across sources, terminology is not yet standardized, and it should be distinguished from broader AI governance (organizational oversight and accountability structures) and from model risk management, though it may interact with both. General AI safety, model performance, and non-agentic AI security concerns are out of scope for this term as defined here.

Why it matters

AI agents differ from earlier AI applications because they do not merely produce outputs for a human to review—they can plan tasks, invoke tools, use memory, and take actions across workflows on behalf of users. This expanded capacity means that a compromised, manipulated, or poorly constrained agent can cause harm directly, for example by executing actions against systems and data it has access to. As a result, the security concern is twofold: protecting the agent itself from threats, and managing the risks that arise from what the agent is permitted to do. Both dimensions are emphasized across current industry and research treatments of the topic.

Because agents act autonomously, the traditional assumption that a human sits between an AI's suggestion and its execution weakens or disappears. Some approaches respond to this by grounding agent security in principles such as ensuring agents have well-defined human controllers, carefully limiting their powers, and keeping their actions observable or constrained. These measures are intended to reduce and manage risk rather than eliminate it, and the field remains early: terminology is not yet standardized and there is no single authoritative definition.

For organizations deploying agentic applications, this area interacts with—but should not be conflated with—broader AI governance (organizational oversight and accountability structures) and model risk management (the identification, measurement, and control of model-related risk). Security of AI agents is a distinct, evolving discipline focused specifically on the threats to, and risks from, autonomous software acting on a user's behalf.

Who it's relevant to

AI governance and risk professionals
Those responsible for organizational oversight of AI systems need to understand how agentic capabilities change the risk picture, since agents can take actions rather than only produce outputs. This term is distinct from broader governance and from model risk management, but it may interact with both, and practitioners should be careful not to collapse the distinctions.
Security engineers and architects
Teams securing agentic applications are responsible for protecting the agent itself and the systems, data, and tools it can access. Their work typically involves applying principles such as well-defined human controllers, carefully limited agent powers, and observable or constrained actions, while recognizing that terminology and practices in this area are not yet standardized.
Data scientists and AI developers building agents
Practitioners designing agents that plan, invoke tools, and use memory need to consider security as part of the design rather than an afterthought, since the agent's permissions and autonomy directly shape the risks it can create. General AI safety, model performance, and non-agentic AI security concerns fall outside this specific term.
Compliance and audit functions
Those evaluating deployed agents should note that this is an evolving area without a single authoritative definition or settled regulatory treatment. Assessing whether appropriate human controls, permission limits, and action constraints are in place can support oversight, but such measures manage and reduce risk rather than eliminate it.

Inside Security of AI Agents

Agent Autonomy and Action Surface
AI agents differ from static models in that they can plan, invoke tools, call external APIs, and take actions with real-world effects. The security scope therefore extends beyond model outputs to the set of actions an agent can initiate, commonly described as its action or attack surface. In many frameworks, the breadth of this surface is treated as a primary driver of inherent risk.
Prompt Injection and Instruction Manipulation
A class of adversarial inputs in which crafted content, whether from a user or from data the agent retrieves, attempts to override the agent's intended instructions. This is typically distinguished from traditional software injection because it exploits the model's handling of natural-language instructions rather than a code-level parser.
Tool and Integration Security
Agents often connect to external tools, databases, and services. Security controls here include scoping tool permissions, authenticating tool calls, and constraining what actions each tool can perform. The concern is that a compromised or manipulated agent may misuse legitimately granted integrations.
Identity, Authentication, and Authorization
Covers how an agent is identified to systems it interacts with and what it is permitted to do. Least-privilege access and separation between the agent's identity and the user's identity are commonly cited controls, though implementation varies by architecture and vendor.
Data Exposure and Confidentiality
Addresses the risk that agents leak sensitive data through outputs, logs, retrieved context, or onward transmission to external tools. This includes both inadvertent disclosure and exfiltration induced by adversarial input.
Monitoring, Logging, and Human Oversight
Encompasses the observability of agent behavior, including audit trails of actions taken, and mechanisms for human review or intervention. Human-in-the-loop or human-on-the-loop controls are commonly described as measures that reduce, but do not eliminate, the risk of harmful autonomous action.
Governance and Risk Management Overlap
Securing AI agents sits at the intersection of AI governance (organizational policies, accountability, and oversight for how agents are deployed) and model risk management (identifying, measuring, monitoring, and controlling risks from the underlying model). The two are related but distinct; agent security draws on both without collapsing them.

Common questions

Answers to the questions practitioners most commonly ask about Security of AI Agents.

Does securing an AI agent simply mean applying the same controls used for traditional model risk management?
Not exactly. Model risk management, as historically framed by guidance such as SR 11-7, focuses on the risks arising from a model's design, use, and outputs, including validation, performance monitoring, and control of model error. Securing an AI agent overlaps with this but is not the same: an agent that can take actions, invoke tools, or interact with external systems introduces attack surfaces and operational security concerns that traditional model risk frameworks were not designed to address in full. Treat agent security as a distinct, though related, discipline that draws on both AI governance structures and model risk practices, rather than collapsing it into either.
If an AI agent has strong governance policies in place, does that mean it is secure?
No. Governance policies establish accountability, oversight, and decision rights for how an agent is built and used, but policies alone do not implement technical security controls, and no set of controls eliminates risk entirely. Governance and security are complementary: governance defines who is responsible and what standards apply, while security involves the concrete measures that reduce the likelihood or impact of compromise. A well-governed agent can still be vulnerable if the underlying technical safeguards are weak, so the two should be assessed separately rather than treating one as a proxy for the other.
What technical controls are commonly considered when securing an AI agent's ability to invoke tools or external systems?
Practitioners typically consider scoping the agent's permissions to the minimum needed, constraining which tools or APIs it can call, and applying authorization checks at each action boundary rather than only at the point of initial access. Additional measures often discussed include input and output validation, isolation or sandboxing of execution environments, and logging of agent actions for later review. The specific controls depend on the agent's capabilities and deployment context, and none of these measures should be treated as guaranteeing security; they are risk-reducing measures whose adequacy depends on the threat environment.
How can an organization monitor an AI agent for security-relevant behavior after deployment?
Monitoring commonly involves recording the agent's actions, tool invocations, and interactions with external systems in a way that supports review and, where feasible, detection of anomalous or unauthorized behavior. This is distinct from monitoring for model performance degradation, which tracks changes in output quality over time; security monitoring focuses on whether the agent is behaving within intended and authorized boundaries. Because agents may operate with some autonomy, organizations often establish escalation paths and human oversight points, though the appropriate design varies by use case and risk level.
Which lines of defense are typically involved in overseeing the security of an AI agent?
In many organizations that use a three-lines model, the first line comprises the teams that build and operate the agent and own the day-to-day controls; the second line provides independent risk and compliance oversight, including challenge of the first line's practices; and the third line, typically internal audit, provides independent assurance over the whole arrangement. Agent security responsibilities are usually distributed across these lines rather than sitting with any single function. The precise allocation depends on the organization's structure, and these roles should be kept distinct rather than blurred.
How should security considerations be reflected when assessing the risk of an AI agent?
Security is generally one input into a broader risk assessment. It can be useful to distinguish inherent risk, the risk before controls are applied, from residual risk, the risk that remains after controls are in place, since security measures aim to reduce but not eliminate residual risk. Assessments often weigh the agent's capabilities, the sensitivity of the systems it can reach, and the potential impact of compromise. Because approaches to agent risk assessment are still evolving, organizations should treat their methods as subject to refinement rather than as settled or universally standardized.

Common misconceptions

Securing the underlying model is sufficient to secure an AI agent.
Model-level safeguards address one part of the picture. Because agents take actions through tools and integrations, security must also cover the action surface, permissions, and external connections. A well-aligned model can still cause harm if its integrations are over-privileged or manipulable.
Prompt injection is just another form of conventional software injection and can be handled with standard input sanitization.
Prompt injection typically exploits how a model interprets natural-language instructions rather than a code parser, and content can arrive indirectly through retrieved data. As commonly described, it resists purely rule-based sanitization and generally requires layered controls rather than a single fix.
Human oversight and access controls eliminate the security risks of autonomous agents.
Oversight and least-privilege access are risk-reducing measures, not guarantees. They manage and constrain harm but do not remove it, particularly where agents act quickly, at scale, or through complex tool chains that make review difficult.

Best practices

Apply least-privilege principles to every tool, integration, and data source an agent can reach, and scope agent identity separately from the invoking user's identity.
Treat all inputs, including data the agent retrieves at runtime, as potentially adversarial, and layer defenses against prompt injection rather than relying on a single sanitization step.
Constrain the agent's action surface explicitly, defining and enforcing which actions require human confirmation and which may execute autonomously.
Maintain audit trails and logging of agent actions, tool calls, and data accessed to support monitoring, incident review, and accountability.
Implement controls against data exposure across outputs, logs, and onward transmission to external tools, with particular attention to sensitive or regulated data.
Coordinate agent security across both AI governance structures and model risk management processes, keeping their responsibilities distinct while ensuring coverage of the risks each addresses.