Scope
This guide focuses on runtime governance controls for agentic AI systems, autonomous agents that make decisions, use tools, and adapt based on feedback. If you're securing systems that can initiate database queries, trigger API calls, or modify their execution paths without human approval, you need runtime governance.
This guide does NOT cover:
- Pre-deployment model validation (see SR 11-7 guidance)
- Static model cards or system documentation
- Governance for traditional supervised learning models with fixed inference paths
Key Concepts and Definitions
Agentic AI System: An AI system that achieves goals through iterative decision-making, tool use, and environmental interaction. Unlike a classification model that outputs a single prediction, an agent executes multi-step workflows.
Runtime Governance: Controls applied during system operation, not just at deployment. Think circuit breakers, not design reviews.
Tool Invocation: When an agent calls external functions, database queries, API requests, file operations, or other model invocations. Each invocation is a potential security boundary.
Execution Context: The complete state of an agent's operation at any moment: conversation history, tools available, permissions granted, and environmental constraints.
Guardrails: Programmatic constraints that prevent specific behaviors regardless of model outputs. These differ from prompt engineering, a guardrail blocks an action; a prompt requests compliance.
Requirements Breakdown
Control Layer 1: Input Validation
Your agent receives instructions from users, other systems, or its own planning loop. Before any tool invocation:
Requirement 1.1: Schema validation for all structured inputs. If your agent accepts JSON task definitions, validate against a strict schema before parsing.
Requirement 1.2: Content filtering at ingestion. Apply controls like injection pattern detection, banned phrase lists, and semantic similarity checks against known attack prompts.
Requirement 1.3: Scope verification. Ensure the user's request falls within the agent's authorized domain. A customer service agent shouldn't execute database administration commands, even if technically capable.
Control Layer 2: Execution Boundaries
Requirement 2.1: Tool allowlisting per context. Maintain a registry of which tools each agent type may invoke. Your SQL agent doesn't need file system access; your document processor doesn't need network calls.
Requirement 2.2: Parameter constraints. If an agent can query a database, constrain which tables, row limits, and query complexity it may use. Don't rely on the model to self-limit.
Requirement 2.3: Rate limiting per tool and per session. Set maximum invocations per minute for each tool type. An agent stuck in a loop shouldn't drain your API quota or overwhelm services.
Requirement 2.4: Execution timeouts. Every tool invocation needs a deadline. If your agent calls an external API, enforce both connection and read timeouts.
Control Layer 3: Output Monitoring
Requirement 3.1: Response inspection before delivery. Check agent outputs for sensitive data patterns (API keys, PII, internal URLs) before returning to users.
Requirement 3.2: Action logging with full context. Log every tool invocation with: timestamp, agent ID, user context, tool name, parameters, response code, and execution duration. You'll need this for incident response and compliance audits.
Requirement 3.3: Anomaly detection on execution patterns. Track statistical norms for each agent type, tools invoked per session, execution time distributions, error rates. Alert on deviations.
Control Layer 4: Circuit Breakers
Requirement 4.1: Error thresholds. If an agent generates N consecutive tool failures, suspend execution and require human review.
Requirement 4.2: Cost gates. Set spending limits per session and per day. An agent that burns through your LLM token budget in one conversation indicates either a bug or an attack.
Requirement 4.3: Confidence thresholds for high-stakes actions. If your agent can approve transactions or modify production data, require explicit confidence scores and block execution below your threshold.
Implementation Guidance
Start with a Reference Architecture
Build a governance proxy that sits between your agent framework and all external tools. This proxy enforces requirements 2.1-2.4 in one place rather than scattered across agent code.
Your proxy should:
- Maintain the tool registry and parameter schemas
- Apply rate limits using a token bucket algorithm
- Log all invocations to your SIEM
- Return structured errors that your agent can handle
Instrument Before You Deploy
Don't retrofit observability. Your agent framework needs telemetry hooks from day one:
- Trace IDs that connect user requests to all downstream tool calls
- Structured logs in a consistent schema (JSON, not free text)
- Metrics exported to your monitoring stack (Prometheus, Datadog, or equivalent)
Test Your Circuit Breakers
Write integration tests that verify your governance controls actually fire:
- Does rate limiting block the 101st request?
- Does your cost gate stop execution at the threshold?
- Can you reproduce and inspect the logged context for any failed invocation?
If you can't reliably trigger your safety controls in testing, they won't work in production.
Establish Review Cadences
Runtime governance isn't set-and-forget. Schedule monthly reviews of:
- Tool invocation patterns (are agents using tools you didn't expect?)
- Error rates and timeout frequencies
- Cost distribution across agent types
- Anomaly detection false positive rates
Common Pitfalls
Pitfall 1: Trusting model outputs for security decisions. Your LLM might generate a SQL query that looks safe. Execute it anyway against a read-only replica with row limits.
Pitfall 2: Logging without retention policies. Agent execution logs contain user inputs and system responses. Apply the same data retention and privacy controls you use for application logs.
Pitfall 3: Inconsistent tool interfaces. If every tool uses different authentication, error handling, and timeout behavior, your governance layer becomes unmaintainable. Standardize.
Pitfall 4: Alerting on everything. You'll drown in noise if every rate limit triggers a page. Distinguish between routine safety controls (log and continue) and genuine incidents (alert and suspend).
Pitfall 5: Skipping the threat model. Before you implement controls, document: What can go wrong? What's the impact? Which controls reduce which risks? Generic checklists don't substitute for context-specific analysis.
Quick Reference Table
| Control Type | Requirement | Implementation Check | Audit Evidence |
|---|---|---|---|
| Input validation | Schema + content filtering | Can you pass malformed JSON? Injection patterns? | Rejected request logs |
| Tool allowlist | Explicit registry per agent type | Can agent A invoke agent B's tools? | Tool registry version history |
| Parameter constraints | Limits on query complexity, row counts, file sizes | Can you request 1M rows? Write to /etc? | Blocked invocation logs |
| Rate limiting | Per-tool, per-session limits | Does request 101 fail? | Rate limit metrics dashboard |
| Execution timeout | Deadline for every tool call | Can you trigger a timeout? | Timeout event logs |
| Output inspection | PII and secret detection | Can you extract an API key from responses? | DLP match logs |
| Action logging | Full context for every invocation | Can you reconstruct a session from logs? | Log retention policy + samples |
| Error thresholds | Circuit breaker on consecutive failures | Does failure N+1 suspend execution? | Circuit breaker state changes |
| Cost gates | Spending limits per session/day | Can you exceed the budget? | Cost tracking dashboard |
| Confidence thresholds | Minimum score for high-stakes actions | Does low-confidence block execution? | Blocked action logs with scores |
You'll know your runtime governance is working when you can answer: "Show me every tool this agent invoked in the last incident" in under five minutes.
