The decision you're facing
Your organization is deploying AI agents with increasing autonomy. These systems can draft communications, approve routine requests, trigger payments, and interact with external parties. The question isn't whether they'll be targeted by social engineering attacks, but when, and whether your governance controls can detect manipulation before damage occurs.
AI agents can be manipulated like business email compromise (BEC) victims. This isn't a theoretical risk. Your team needs to decide now: which AI systems get direct communication authority, and what controls must be in place before you grant that access.
Key factors that affect your choice
Three factors determine your path forward:
Agent autonomy level. Does the AI system execute actions independently, or does it always route decisions through human review? An agent that can send invoices, approve vendor changes, or modify access permissions operates at a fundamentally different risk tier than one that drafts recommendations for human approval.
Communication surface area. Does the agent process inbound emails, API calls, or chat messages from external parties? Does it initiate outbound communications? The broader the interaction surface, the more attack vectors you're exposing.
Consequence severity. What's the worst outcome if the agent is successfully manipulated? A compromised marketing chatbot creates reputational risk. A compromised procurement agent can authorize fraudulent payments or vendor substitutions.
Map your AI systems against these three dimensions before you proceed. If you're scoring high on autonomy and consequence severity, you're in Path A territory. If communication exposure is your primary concern but consequences remain contained, you're likely facing Path B decisions.
Path A: High-autonomy agents with financial or access authority
Choose this path when your AI agent can execute transactions, modify credentials, approve payments, or change system configurations without human confirmation.
You're implementing what SR 11-7 would recognize as a high-risk model requiring enhanced validation controls. Your governance framework must include:
Pre-deployment validation. Red Teaming specifically designed for social engineering scenarios. Your validation team should attempt prompt injection attacks, context manipulation, and authority impersonation. Document what breaks the agent and at what threshold.
Dual authorization for sensitive actions. Even if the AI can technically execute alone, require human confirmation for any action above defined thresholds. Define those thresholds by consequence, not just dollar amount, a $500 vendor address change can be more damaging than a $5,000 approved invoice to a known supplier.
Immutable audit trails. Every input the agent receives and every decision it makes must be logged with sufficient context to reconstruct the decision path. ISO/IEC 42001's Clause 6.1.3 requirement for risk treatment applies here. You need evidence that you identified the social engineering risk and implemented specific controls.
Anomaly detection on agent behavior. If your procurement agent suddenly approves three vendor changes in a week after averaging one per quarter, something's wrong. Build behavioral baselines and alert on deviations before the agent executes.
Communication authentication. Implement sender verification that the AI can't override. If your agent processes email, it must validate sender domains, DMARC alignment, and known correspondent patterns before treating a request as legitimate.
Consider a team deploying an AI agent with invoice approval authority. Before granting that access, you'd require successful completion of adversarial testing where researchers attempt to manipulate the agent into approving fraudulent invoices through carefully crafted emails. You'd set hard limits, perhaps the agent can approve up to $10,000 per vendor per month, with anything above requiring human review. You'd implement real-time monitoring that flags any approval to a new vendor or changed bank account.
Path B: Communication-exposed agents with limited execution authority
Choose this path when your AI agent interacts with external parties but can't execute consequential actions independently, customer service bots, scheduling assistants, information retrieval systems.
The risk isn't financial fraud; it's information disclosure, reputational damage, or using the agent as a reconnaissance tool for later attacks.
Your governance framework should include:
Information classification boundaries. Define explicitly what data categories the agent can access and disclose. A customer service agent might retrieve order status but never payment card details. Enforce these boundaries at the data access layer, not just through prompt engineering.
Output filtering. Implement content screening that blocks the agent from disclosing sensitive patterns, internal system names, employee details, security configurations, even if a manipulated prompt attempts to extract them.
Conversation monitoring. Unlike Path A's transaction focus, you're monitoring for reconnaissance attempts. Flag conversations where external parties probe system capabilities, ask about internal processes, or attempt to establish the agent's authority boundaries.
Graceful degradation to human handoff. When the agent detects potential manipulation, unusual question patterns, requests outside its scope, attempts to override instructions, it should route to human review rather than attempt to handle the interaction.
Session limits. Sophisticated social engineering often involves building rapport over multiple interactions. Implement conversation length limits and periodic session resets that prevent attackers from incrementally expanding the agent's disclosed information.
A scheduling assistant that can view calendars and book meetings needs different controls than an invoice approver. You'd restrict its data access to calendar availability only, implement filtering that prevents it from disclosing meeting titles or attendee lists, and program automatic escalation when someone asks about executive availability patterns or attempts to modify existing meetings.
Path C: Internal-only agents with no external communication
Choose this path when your AI agent operates entirely within your security perimeter, processing only authenticated internal requests.
You're not immune to social engineering, internal actors can manipulate these agents too, but your attack surface is dramatically smaller. Your governance framework focuses on:
Access control integration. The agent should respect your existing identity and access management policies. If a user can't manually access a dataset, the agent shouldn't retrieve it on their behalf.
Prompt injection defenses. Internal users might attempt to manipulate the agent into bypassing controls through carefully crafted prompts. Implement input validation and instruction hierarchy that prevents user inputs from overriding system-level constraints.
Usage monitoring. Track which users interact with the agent and what actions they request. Anomalous patterns, an employee suddenly asking the agent to retrieve data outside their normal scope, warrant investigation.
The validation burden is lighter than Path A, but you still need evidence that you've tested manipulation scenarios and implemented appropriate controls.
Summary matrix
| Decision Factor | Path A (High-Risk) | Path B (Communication-Exposed) | Path C (Internal) |
|---|---|---|---|
| Autonomy Level | Executes transactions independently | Provides information, limited actions | Retrieves/processes data |
| Communication Surface | External, bidirectional | External, constrained | Internal only |
| Consequence Severity | Financial loss, unauthorized access | Information disclosure, reputation | Policy violation, data misuse |
| Primary Controls | Dual authorization, transaction limits, red teaming | Output filtering, conversation monitoring, handoff protocols | Access control integration, prompt defenses |
| Validation Intensity | Enhanced (SR 11-7 equivalent) | Moderate, scenario-focused | Standard, control-focused |
| Monitoring Focus | Behavioral anomalies, transaction patterns | Reconnaissance attempts, information probing | Access patterns, privilege escalation |
Your AI agents will be targeted. The question is whether your governance framework can detect and block the manipulation before it succeeds. Choose your path based on consequence severity, not deployment convenience.




