An enterprise customer onboarding agent pulls data from three internal APIs, generates a compliance summary, and routes the output to a second agent for review. The retrieval step returns stale data. The summary looks plausible. Latency is normal. No error fires. The customer receives an inaccurate compliance document. Nobody knows until a regulator flags it weeks later.
This isn't a hypothetical. It's the failure mode your observability stack wasn't built to catch.
What Happened
An autonomous AI agent executing a multi-step customer onboarding workflow produced an inaccurate compliance document that reached the customer and triggered regulatory scrutiny. The agent operated within expected latency bounds, generated no error signals, and completed the task as designed. The failure occurred during data retrieval: the agent pulled stale information from an internal API, synthesized it into a coherent but factually incorrect summary, and passed it to a downstream agent for review. Both agents performed their assigned functions. The output appeared valid. The control environment detected nothing unusual.
The incident surfaced only when external regulators identified the compliance error weeks after delivery.
Timeline
Initial execution: Agent receives customer onboarding task and initiates multi-step workflow.
Retrieval phase: Agent queries three internal APIs for customer data. One API returns outdated records. No error is logged because the API responded successfully with well-formed data.
Synthesis phase: Agent generates compliance summary based on retrieved data. The summary is coherent and structurally correct. Quality metrics show normal scores.
Review phase: Second agent reviews the output against format and completeness criteria. The document passes review because the error is factual, not structural.
Delivery: Customer receives compliance document containing inaccurate information.
Detection: Regulatory body flags the error during routine review, weeks after delivery. Internal investigation begins.
Which Controls Failed or Were Missing
Trace-level visibility across decision chains: The observability stack captured latency, token usage, and error rates. It did not trace the agent's decision path from retrieval ranking through synthesis to final output. When the investigation began, teams could see that the workflow completed successfully. They could not reconstruct which data sources the agent prioritized, why it selected stale records, or how the error propagated through subsequent steps.
Production evaluation of intermediate outputs: Pre-deployment testing validated the agent's performance on curated datasets. No control evaluated the factual accuracy of retrieval results or synthesis outputs in production. The agent's behavior in the live environment diverged from its behavior in testing because real-world API responses introduced variance the test suite never replicated.
Behavioral monitoring for drift: The system monitored infrastructure health and response times. It did not track changes in the agent's retrieval patterns, tool utilization, or data source preferences over time. If the agent had gradually shifted toward older API endpoints or changed its ranking logic, no alert would have fired.
Data freshness validation: The retrieval layer returned data without timestamp checks or staleness indicators. The agent had no mechanism to verify that retrieved records were current. The downstream review agent evaluated format and structure, not factual accuracy or data recency.
What the Relevant Standard Requires
ISO/IEC 42001 (AI Management System): Annex A Control 6.2.6 requires organizations to "ensure that data used by the AI system are current and relevant to the purpose of the AI system." This control directly addresses data quality in operational AI systems. The incident violated this requirement because the system lacked controls to verify data currency before synthesis.
NIST AI RMF (Measure function): The Measure function requires organizations to "assess AI system performance and impacts using metrics and methodologies that are appropriate for the context and type of AI system." For agentic systems, appropriate measurement includes trace-level evaluation of decision chains, not just aggregate output metrics. The incident demonstrates what happens when measurement stops at the boundary of a single inference call.
SR 11-7 (Model Risk Management): Section III.B.2 requires ongoing monitoring to detect "any changes in product performance or new risk that has emerged." For autonomous agents, this means monitoring behavioral drift: changes in tool usage, retrieval patterns, or decision paths that signal degraded performance before outputs fail. The incident went undetected because monitoring focused on infrastructure metrics rather than agent behavior.
ISO/IEC 5338 (AI Lifecycle Processes): The monitoring and continual learning process (clause 6.3.11) requires "monitoring of the AI system's operation to detect anomalies, performance degradation, or other issues." Effective monitoring for agentic systems must extend beyond error rates to include decision quality, reasoning coherence, and alignment across multi-step workflows.
Lessons and Action Items for Your Team
Instrument for distributed tracing, not just metrics. Deploy OpenTelemetry-based instrumentation that captures LLM-specific spans across your agent's full execution tree. You need visibility into tool calls, retrieval actions, reasoning chains, and handoffs between agents. If your current platform shows latency and error rates but cannot trace a bad output back through the decision chain that produced it, you're monitoring infrastructure, not agent behavior.
Build production evaluation into your control environment. Pre-deployment testing is necessary but insufficient. Implement continuous evaluation that measures faithfulness, groundedness, and relevance for intermediate outputs in production. For the onboarding agent scenario, this means evaluating retrieval quality and synthesis accuracy in real time, not just validating the final document format.
Define behavioral baselines for each agent. Document expected patterns: which tools the agent calls, what data sources it accesses, how many steps it takes per task, what its typical retrieval ranking looks like. Monitor for deviations. A 40% shift in tool utilization or a change in data source preference is a signal, not noise. Without a baseline, drift is invisible.
Validate data freshness at the retrieval layer. If your agent pulls data from internal APIs, implement timestamp checks and staleness thresholds. The retrieval layer should reject or flag outdated records before they enter the synthesis phase. This is a control gap you can close without rearchitecting your observability stack.
Extend review controls beyond format checks. If you're using a second agent to review outputs, configure it to evaluate factual accuracy and data currency, not just structural correctness. The review agent in this incident passed a document that met format requirements but failed on substance. That's a scope problem, not a capability problem.
Organizations moving agentic AI from pilot to production are building visibility infrastructure that matches the complexity of autonomous decision chains. If your observability stack was designed for request-response patterns, you're flying blind. The next silent failure is already in your pipeline.
