Scope of This Guide
This guide covers governance requirements for agentic AI systems in banking, where AI components make decisions, invoke tools, and execute multi-step processes with minimal human intervention. It's intended for governance teams transitioning from traditional predictive models to autonomous AI agents that interact with systems, retrieve data, and complete tasks.
You'll find requirement mappings, implementation steps, and a reference table for design reviews and validation planning. This guide assumes familiarity with SR 11-7 and ISO/IEC 42001; it extends those frameworks to cover agency-specific controls.
Key Concepts and Definitions
Agentic AI System: An AI system that pursues goals through multi-step reasoning, tool invocation, and autonomous decision-making. Unlike traditional models that output a score or classification, agentic systems plan sequences of actions and adapt based on intermediate results.
Tool Invocation: The ability of an AI agent to call external functions, APIs, or data sources during execution. Examples include querying databases, triggering payment workflows, or generating documents.
Execution Trace: A complete record of an agent's decision path, including reasoning steps, tool calls, intermediate outputs, and final actions. Required for accountability and audit.
Human-in-the-Loop (HITL) Control Point: A mandatory approval gate where a human reviews and authorizes an agent's proposed action before execution. Different from monitoring; this is an active intervention requirement.
Rollback Capability: The technical ability to reverse or undo an agent's actions if errors or policy violations are detected post-execution.
Requirements Breakdown
Governance Structure (ISO/IEC 42001 § 5.3, SR 11-7 § II.A)
Traditional model governance assumes a single artifact, a trained model, that you validate once and monitor continuously. Agentic systems break this assumption. You're governing a workflow composed of:
- A foundation model or ensemble of models
- Retrieval mechanisms and knowledge bases
- Tool libraries and API integrations
- Orchestration logic that determines action sequences
Requirement: Establish governance ownership at the workflow level, not just the model level. Your AI Management System must define accountability for the entire agent, including non-model components.
Practical Step: Assign a single Model Owner (or Agent Owner) responsible for the complete workflow. This person coordinates across data engineering (retrieval), application teams (tool integration), and model risk (validation). Document this assignment in your model inventory with explicit scope boundaries.
Traceability and Explainability (EU AI Act Article 13, NIST AI RMF Measure 2.7)
When an agent executes a multi-step workflow, you need visibility into every decision point.
Requirement: Implement execution tracing that captures:
- Input prompts and contextual data retrieved
- Reasoning steps or chain-of-thought outputs
- Each tool invocation with parameters and return values
- Final action and system state changes
Practical Step: Build structured logging into your agent orchestration layer. Use a schema that links each action to its triggering reasoning step. Store traces in immutable audit logs with retention aligned to your records management policy (typically 7 years for financial services under SR 11-7).
Common Gap: Many teams log final outputs but skip intermediate reasoning. This makes root cause analysis impossible when an agent produces an incorrect result.
Validation Evidence (SR 11-7 § III.B, ISO/IEC 42001 Annex A.6.1.5)
You can't validate an agentic workflow the same way you validate a credit scoring model.
Requirement: Validation evidence must cover:
- Accuracy of individual model components
- Correctness of tool invocation logic
- Appropriateness of action selection given context
- Boundary behavior (what happens when the agent encounters novel situations)
Practical Step: Design test scenarios that exercise complete workflows, not just model predictions. Include adversarial cases where the agent should refuse to act or escalate to a human. Document expected behavior for out-of-scope requests.
Example test case structure:
- Scenario: Customer requests a wire transfer exceeding daily limits
- Expected Reasoning: Agent identifies limit violation, explains constraint, offers alternative (split transfer)
- Expected Tool Calls: Account lookup, limit check, no transfer execution
- Pass Criteria: Agent does not invoke transfer API; provides accurate explanation
Human Oversight Controls (EU AI Act Article 14, NIST AI RMF Govern 1.5)
Autonomous doesn't mean unsupervised.
Requirement: Define HITL Control Points based on risk tiering. High-impact actions (account closures, large transfers, credit decisions) require human approval before execution. Lower-risk actions (information retrieval, routine inquiries) can proceed with post-execution monitoring.
Practical Step: Implement a control matrix mapping action types to oversight requirements:
| Action Type | Risk Tier | Control Type | Approval SLA |
|---|---|---|---|
| Data retrieval | Low | Post-execution review | N/A |
| Document generation | Medium | Sampling review (10%) | 24 hours |
| Payment initiation | High | Pre-execution approval | Real-time |
| Account modification | High | Pre-execution approval | Real-time |
Configure your orchestration layer to enforce these gates programmatically.
Rollback and Remediation (ISO/IEC 42001 Annex A.6.1.10)
When an agent makes a mistake, you need technical and procedural mechanisms to fix it.
Requirement: Design rollback procedures for each consequential action type. Document what can be automatically reversed versus what requires manual intervention.
Practical Step: Maintain a rollback playbook that includes:
- Automated rollback scripts for reversible actions
- Escalation paths for actions requiring business judgment
- Customer communication templates
- Incident documentation requirements
Test rollback procedures during validation, not after an incident.
Implementation Guidance
Start with Constrained Agents
Don't deploy a fully autonomous agent on day one. Begin with narrow use cases where the action space is limited and consequences are reversible.
Consider a team building a customer service agent. Phase 1 might allow the agent to retrieve account information and answer questions, but not modify records. Phase 2 adds the ability to update mailing addresses after human review. Phase 3 enables autonomous address updates for low-risk changes (same city, verified through additional authentication).
This staged approach lets you refine traceability, validation methods, and oversight controls before expanding agent capabilities.
Integrate with Existing Model Risk Frameworks
Your agentic AI governance doesn't replace SR 11-7 or ISO/IEC 42001, it extends them.
Map agent components to your existing model inventory. The foundation model powering the agent's reasoning is a model subject to validation. The retrieval system is a data dependency. The orchestration logic is operational risk.
Reuse your model validation templates, adding sections for tool invocation testing and multi-step scenario coverage. This maintains consistency across your model risk program while addressing agent-specific requirements.
Build Observability Before Deployment
You can't govern what you can't see. Implement monitoring that surfaces:
- Action success/failure rates by type
- Reasoning quality (measured through human review sampling)
- Tool invocation errors
- Escalation frequency
- Execution time distributions
Set alerts for anomalies: a sudden spike in tool errors, increased escalation rates, or reasoning outputs that match known failure patterns.
Common Pitfalls
Pitfall 1: Treating the Foundation Model as the Entire System You validate the language model's general capabilities but ignore the workflow logic that determines when and how it's invoked. Result: the model performs well in isolation but makes poor decisions in production because the orchestration layer provides inadequate context.
Pitfall 2: Insufficient Execution Trace Granularity You log inputs and final outputs but skip the reasoning chain. When an agent produces an incorrect result, you can't determine whether the error originated in retrieval, reasoning, or tool execution.
Pitfall 3: Assuming Determinism Traditional models produce consistent outputs for identical inputs (given fixed parameters). Agentic systems often don't, the same query might trigger different tool calls or reasoning paths based on retrieved context or stochastic generation. Your validation approach must account for this variability through scenario-based testing, not just input-output pairs.
Pitfall 4: Overlooking Downstream Dependencies An agent invokes an API that modifies a record in a system of record. That system has its own validation rules, access controls, and audit requirements. You need end-to-end testing that verifies the agent respects these constraints, not just that it formats API calls correctly.
Quick Reference Table
| Governance Requirement | Traditional Model | Agentic AI System | Key Difference |
|---|---|---|---|
| Validation Scope | Model artifact, training data | Workflow (model + tools + orchestration) | Multi-component system |
| Traceability | Input → output mapping | Execution trace with reasoning steps | Multi-step decision path |
| Testing Approach | Input-output pairs, performance metrics | End-to-end scenarios, boundary cases | Workflow completeness |
| Human Oversight | Review predictions before business use | HITL gates at high-risk action points | Action-level controls |
| Monitoring Focus | Prediction accuracy, drift | Action outcomes, tool errors, escalation rates | Operational behavior |
| Rollback Mechanism | Model version revert | Action-specific remediation procedures | Per-action reversibility |
| Documentation (SR 11-7) | Model Card, validation report | System Card + workflow diagrams + tool inventory | Expanded scope |
| Accountability | Model Owner | Agent Owner (coordinates across teams) | Cross-functional ownership |
Bookmark this: When you're designing governance for your next agentic AI deployment, use this table to identify gaps between your current model risk framework and the controls you'll need for autonomous workflows.
