The Challenge
Your travel-booking agent confidently recommended hotels, but customers ended up at the wrong properties. Despite all logs showing green, the agent's responses were flawed, and traditional monitoring tools missed it. The problem was only discovered after shifting from span-level monitoring to full-session tracing. The agent received a 200 OK status with a nearly empty results array, then recommended a more expensive, less convenient hotel based on false availability data. This silent failure occurred because the transport layer succeeded, but the payload was unusable. Standard safety evaluations didn't catch this issue.
The Environment and Constraints
Your team faced three constraints that complicated discovery. First, your monitoring treated each agent step as an isolated event, missing failures in the causal chain between steps. Second, cost pressures led to aggressive down-sampling, discarding the rare silent-failure sessions you needed to study. Third, without session-level replay capability, spans were disconnected events, making it impossible to reconstruct the agent's workflow.
The Approach Taken
Your team implemented a four-step workflow to identify failure patterns:
Step one: capture full-fidelity, content-level traces. Record every LLM and tool call as a span with inputs, outputs, arguments, latency, and a shared session ID. This shift from metadata-only spans to content-level spans captured what the agent actually sent and received. Add a validation check for any 200 status with no usable data:
def classify_span(span):
body = span["response"]
empty = not body.get("results")
partial = body.get("partial") is True
if span["status_code"] == 200 and (empty or partial):
return "silent_failure"
return "ok"
Step two: evaluate every trace, not a sample. Score 100% of traffic for groundedness, faithfulness, relevance, and toxicity. Use Fiddler Centor Models to make full-population scoring practical.
Step three: cluster failures by signature. Group spans by error type, tool, step, and behavior to condense thousands of log lines into a few ranked issues. Each cluster includes a frequency count and a representative trace for replay.
Step four: add guardrails at boundaries. Implement pre-LLM guardrails to intercept inputs and post-execution guardrails to inspect outputs. Redact PII and PHI by default to protect sensitive data.
Results and Metrics
The workflow identified the hotel-search silent failure as a recurring issue. The agent now flags any 200 response with an empty or partial results array, prompting the user for clarification instead of fabricating availability. The fix was verified by running the same clustering query against the next day's traffic, reducing the silent-failure signature to near zero. You can now map any production failure back to a specific step, tool, or handoff and verify fixes against live traffic.
What They Would Do Differently
Your team regrets not enabling content-level tracing sooner. Concerns about storage costs and PII exposure were addressable: redaction protects sensitive data, and running evaluators in your own environment eliminates external costs. They also wish they'd implemented session replay from the start. Without session IDs, early debugging was tedious. With session replay, root-cause analysis became straightforward.
Takeaways for Your Team
Discovery is a governance precondition, not a debugging afterthought. You can't grant an agent more independence than your ability to see what it did. Observability must come before autonomy.
Log-based monitoring catches individual errors, not failure patterns. Silent failures emerge from the causal chain between steps. If your monitoring treats each span as isolated, you're blind to critical patterns.
Down-sampling defeats the purpose when hunting rare failures. The sessions you drop are the ones you need to study. Full-population evaluation is feasible when you run evaluators in your own environment.
Content-level tracing without redaction writes PII into your trace store. Redact sensitive fields at capture time. Your tool endpoints can pull personally identifiable information directly into agent context.
Detecting and enforcing checks against patterns increases autonomy. Oversight and independence grow together. The task isn't writing more alerts; it's closing the loop from a discovered pattern to an enforced check, before another traveler books the wrong room.
