The Conventional Wisdom
If your team has built a model risk framework, you might think you're ready to govern AI agents. You'd assume that deploying existing guardrails, running standard validation protocols, and monitoring for drift will suffice. The infrastructure designed for predictive models should scale to agentic systems with just minor tweaks.
This belief is widespread. Many teams treat agent governance as a guardrail issue: add prompt injection filters, run toxicity checks, validate outputs against ground truth, and you're set.
Why This View Is Incomplete
Guardrails catch what they're designed to catch. They don't address what happens between them.
Agentic systems introduce a coordination problem that traditional monitoring wasn't built to solve. When an agent calls multiple tools, queries knowledge bases, and routes through several model invocations to answer one question, your guardrails only see the final output. They miss session handoffs, tool selection logic, and the probabilistic routing that led to that output.
David Kenny uses a direct analogy: a control plane is like "wearables inside your agents." Wearables show real-time activity across the full chain. Guardrails check discrete outputs at fixed points. This difference is crucial because agent failures often occur in the coordination layer, not the final response.
Consider a compliance workflow where an agent must verify that a customer meets three regulatory criteria before approving a transaction. Your guardrails check the final approval message for toxic language and policy violations. They don't verify that the agent consulted all three criteria, used the correct data sources, or followed the documented decision path. The output looks clean, but the process was flawed.
This oversight leads to "rogue behavior": teams scale autonomy faster than they scale control.
The Evidence
The problem appears in telemetry gaps. Traditional model monitoring tracks input distribution, prediction confidence, and output quality. It doesn't track which model handled which subtask, whether the agent used deterministic logic when a known answer existed, or how many redundant calls inflated your latency and cost.
Kenny's advice on timing was clear: build the control plane early. His team at Nielsen learned this firsthand. "Having the data along the way really helps. If we'd had the data a year earlier, we probably would've gotten to market faster and more effectively."
The control plane he describes enforces three operational pillars: truth, safety, and economics. Guardrails address part of safety. They don't address truth (did the agent return the correct answer using the correct process?) or economics (did it waste your budget with unnecessary LLM calls?).
For truth, the issue is fitness for purpose. When a customer asks for a specific TV rating, there's one correct answer. Using an LLM to generate a probabilistic response wastes resources and introduces hallucination risk. Kenny warns: "People who use LLMs as search engines to get a specific answer are wasting a lot of energy and costs."
For safety, the issue is timing. Guardrails run at inference boundaries. They catch toxic outputs after the model generates them. They don't prevent an agent from following a malicious instruction during a multi-step workflow. Kenny emphasized this for Nielsen's deployment: "We needed to make sure agents didn't jailbreak into giving instructions that would be unacceptable to our system."
For economics, the issue is visibility. Without session-level cost tracking, you can't identify which workflows are burning budget or where latency is degrading the user experience.
What to Do Instead
Build a control plane that instruments the coordination layer, not just the output layer.
Start with telemetry that captures:
- Which model or logic path handled each subtask
- Whether a deterministic answer path existed and whether the agent used it
- Session-level cost and latency across all model calls and tool invocations
- Agent decision points that led to the final output
Then enforce fit-for-purpose routing. If the answer is known, use deterministic logic or retrieval and reserve the LLM for language formatting. If approximation is acceptable, use predictive ML with calibrated uncertainty. If creative generation is required, use an LLM but constrain it with policies and real-time safety checks.
This is compound AI in practice, and your control plane makes it auditable.
For safety, layer real-time enforcement on top of guardrails. Monitor for prompt injection, policy violations, and adversarial behavior during the session, not after it completes. Your control plane should support both prevention (guardrails) and investigation (observability).
For economics, set budget limits at the session level and surface inefficiencies like redundant model calls or unnecessary tool use. If a workflow consistently exceeds latency targets, your telemetry should pinpoint which step is the bottleneck.
Finally, design for auditability. Kenny's vision for "generally accepted trust principles" is practical, not aspirational. He draws the parallel to accounting standards: "The reason the stock market still works, even with all the craziness in the world, is because we have generally accepted accounting principles. In an AI world, how can you get to generally accepted trust principles?"
Your control plane captures the evidence needed to validate those principles. Define what "misbehavior" means in your domain across truth, safety, and economics. Log model routing and key decisions. Retain traces that support root cause analysis across the agentic hierarchy. Apply sampling workflows that compare outputs to trusted data sources.
When the Conventional Wisdom Is Right
Guardrails still matter. They're your first line of defense for toxicity, policy violations, and known attack patterns. If you're deploying a single-turn chatbot with no tool use and no session state, guardrails may be sufficient.
The conventional wisdom breaks down when autonomy increases. Multi-step workflows, tool-calling agents, and hierarchical agent systems require coordination-layer visibility that guardrails don't provide.
If you're scaling agentic systems to production, your governance infrastructure needs to scale with them. That means building the control plane early, not retrofitting it after launch. The gap between "healthy system metrics" and "customers experiencing incorrect answers" is where rogue behavior lives. Close it with telemetry that exposes what's happening between the guardrails.
