What Changed
Your governance model was built for deterministic systems, but agentic applications just broke it.
Systems where large language models (LLMs) dynamically direct their own processes and tool usage, known as "agentic systems," require real-time intervention, not just post-deployment review. The shift from workflows (predefined code paths) to agents (dynamic, autonomous decision-making) means your model risk framework can no longer rely solely on validation evidence. You need a control plane that monitors, scores, and halts execution before unsafe outputs reach production.
This isn't an incremental change. It's a governance architecture problem.
Key Findings
1. Failure modes multiply across four distinct layers
Agentic systems fail in ways traditional ML models don't. Data access issues (retrieval errors, indexing failures, incomplete knowledge graphs), prompting failures (insufficient context, misleading instructions), model-level errors (hallucinations, brand-unsafe content), and security risks (PII leakage, prompt injection attacks) create compounding risk. A single agent failure can cascade through multi-agent workflows, making root cause analysis exponentially harder.
2. Post-market monitoring isn't enough
Your ISO/IEC 42001 Annex A controls assume you can detect drift after the fact. Agentic systems require in-flight guardrails that evaluate inputs and outputs in real time, rerouting logic or blocking execution when risk thresholds are breached. OpenAI recommends implementing guardrails across multiple dimensions as foundational architecture.
3. Span-level telemetry becomes mandatory
OpenTelemetry has standardized how AI telemetry data is structured across spans, traces, and sessions. Without this granularity, you can't trace a failure from triggered alert through agent hierarchy to the specific span that broke. Your logs must capture inputs, outputs, latency, model version, and agent ID at every decision point.
4. Application-level metrics replace model-level KPIs
Monitoring dozens of metrics at the span level for every agent operation doesn't scale. Instead, define a focused set of high-level indicators: multi-dimensional safety (illegal, hateful, harassing content), toxicity, PII leakage, faithfulness (grounding in source data), conciseness, coherence, and answer relevance. These become your dashboard, not validation evidence footnotes.
5. Guardrails are governance controls
A guardrail that scores outputs for brand safety or data privacy isn't just a technical feature; it's a control that belongs in your AI Management System. Depending on criticality, guardrails should trigger dynamic actions: reroute execution, block unsafe outputs, or escalate to human review. This control plane architecture must map to your risk tiering and materiality thresholds under NIST AI RMF or SR 11-7.
What This Means for Your Team
Your model inventory just became an agent inventory. Every autonomous unit needs its own risk profile, its own set of guardrails, and its own monitoring strategy. Multi-agent workflows require orchestration-level observability, not just model-level validation.
If you're running agents in AWS Bedrock, your logs sit in Amazon S3. If you're using LangGraph, you have direct access to structured telemetry from orchestration pipelines. Either way, you need to ingest, parse, and act on that data in real time, not quarterly.
Your Technical Documentation (Annex IV) under the EU AI Act or your model validation evidence under SR 11-7 must now include: guardrail thresholds, alert trigger logic, escalation paths, and evidence that your control plane actually halts unsafe execution. "We monitor for drift" isn't sufficient. You need to demonstrate real-time intervention capability.
Action Items by Priority
Immediate (next 30 days):
- Audit your current agent deployments. Identify which systems meet Anthropic's definition of agentic (LLMs dynamically directing their own processes).
- Map each agent's failure modes across the four layers: data access, prompting, model-level, and security.
- Implement OpenTelemetry instrumentation if you haven't already. Without standardized span-level telemetry, you can't debug multi-agent failures.
Near-term (60-90 days):
- Define application-level metrics tied to your organization's risk appetite. Start with multi-dimensional safety, PII detection, and faithfulness. Set thresholds that trigger alerts, not just log entries.
- Deploy guardrails for your highest-risk agents. If brand safety or data privacy is critical, implement real-time scoring that can block outputs before they reach users.
- Establish a control plane architecture that maps guardrails to your existing governance framework. Document how guardrail triggers escalate to human review or halt execution.
Ongoing:
- Build a feedback loop between alert triggers and guardrail tuning. When an alert fires, drill down from metric to agent to session to span. Use root cause analysis to refine thresholds and reduce false positives.
- Update your AI System Impact Assessment (ISO/IEC 42005) or AI RMF Profile to reflect agentic architectures. Your risk tiering must account for dynamic, autonomous decision-making.
- Train your model risk and assurance teams on span-level debugging. They need to understand how to trace a failure through multi-agent workflows, not just validate a single model's performance.
