The Challenge
Your team at a mid-tier asset management firm uses a three-agent pipeline for automated research reporting. Agent A retrieves market data, Agent B performs quantitative analysis, and Agent C generates client-facing reports. One day, the final report included a fabricated quarterly earnings figure for a holding that didn't exist in the source data.
You checked the logs. Every agent showed successful completion. No errors, no timeouts, no guardrail violations.
The failure happened between Agent B and Agent C. Agent B's output included a context window of 14,000 tokens, but Agent C's input context was truncated to 8,000 tokens. This truncation dropped a table of verified data points, leaving behind a partial summary. Agent C filled in the missing data with a fabricated figure.
This highlights a blind spot in multi-agent systems: each agent is a black box, and traditional logging captures inputs and outputs at the agent level but misses what happens at the boundary.
The Environment and Constraints
Your team operates in a heterogeneous stack. One group builds on LangGraph. Another uses a custom orchestrator. A third integrates with a vendor-hosted agent. Framework-native tracing can't span these boundaries.
Three common patterns failed:
Flat structured logging. Each agent emitted JSON logs with timestamps, input/output payloads, and status codes. But flat logs have no parent-child relationships. When a failure surfaced in Agent C, there was no structured way to trace backward through Agent B to Agent A. Engineers manually correlated timestamps.
Framework-native tracing. Tools like LangGraph and Amazon Bedrock provide built-in tracing, but only for agents within their frameworks. The team's heterogeneous stack made this approach non-viable.
Generic APM tools. Application performance monitoring platforms modeled agents as microservices and applied distributed traces from traditional software. This captured latency, throughput, and error rates but missed semantic context. An APM tool could report that Agent B sent a payload to Agent C in 47ms. It couldn't report that the payload was missing a critical data table or that Agent B's confidence score fell below the policy threshold.
None of these patterns captured why an agent handed off, what context was dropped at the boundary, or whether guardrail state transferred to the receiving agent.
The Approach Taken
Your team implemented structured handoff tracing using OpenTelemetry's semantic conventions for GenAI agent spans. Each handoff now captures five elements:
Trace ID propagation. Every handoff carries a trace ID following the W3C Trace Context specification, linking the handoff span to the parent trace and enabling reconstruction of the full agent hierarchy.
Handoff payload schema. Structured fields for sender identity, receiver identity, trigger condition, context snapshot, and reasoning summary.
Decision metadata. The sending agent's confidence score, policy evaluation results, and tool call outcomes that informed the handoff decision.
Context diff. A comparison of what the receiving agent actually got versus what the sending agent had available. This field diagnoses silent context truncation.
Guardrail state. Which policies were active on the sending agent, which fired during execution, and which were inherited by the receiving agent.
The implementation tracks context_keys_dropped as an explicit span attribute. When Agent B's output exceeds Agent C's context window, the handoff span records exactly which keys were lost. OpenTelemetry's context propagation mechanism ensures the trace ID flows through every handoff in the chain.
Your team also set explicit max_context_tokens on every handoff and configured warnings when truncation occurs. They tracked guardrails_inherited as a span attribute and set alerts when the count drops to zero on a receiving agent.
Results and Metrics
With proper handoff tracing in place, you can now query the trace hierarchy when a multi-agent pipeline produces an unexpected output. Instead of grepping through logs, you query: show every handoff span where context_keys_dropped is non-empty, filtered by the session ID of the failed request. Root causes surface in seconds, not hours.
For the original incident, the trace hierarchy now shows that Agent B's output was 14,000 tokens. The handoff span records that Agent C received 8,000. The context_keys_dropped attribute lists the verified data table. The guardrail state shows that Agent B's faithfulness policy didn't propagate to Agent C.
Your team runs continuous handoff quality monitoring to surface degradation at the agent-interaction layer. They track handoff faithfulness scores over time and alert when the quality of context transfer degrades. This proved critical when upstream agents are updated independently, a model swap in Agent A can silently change the structure of its output, breaking Agent B's assumptions about the payload schema.
What They Would Do Differently
Your team identified evaluation latency at handoff boundaries as an unexpected bottleneck. Running faithfulness or groundedness checks at every handoff created latency issues when those evaluations called external APIs. Each round-trip added 200 to 500ms per handoff. In a five-agent pipeline, that's 1 to 2.5 seconds of evaluation overhead per request. They've since moved to Fiddler Centor Models, which run evaluation in-environment with no external API calls and under 100ms response time.
They also wish they'd set a max_handoff_depth parameter earlier. Without it, the pipeline occasionally entered circular handoff loops, Agent A hands off to Agent B, which hands off to Agent C, which hands back to Agent A. These loops burned compute and generated thousands of spans. They now set a ceiling (typically 5 to 10 handoffs per session) and terminate the trace when the threshold is reached.
One area remains unsolved: tracing asynchronous agent interactions where the response path differs from the request path. Event-driven architectures break the parent-child span model. When Agent A publishes a message to a queue and Agent B consumes it minutes later, the causal link between the two agents is temporal, not structural.
Takeaways for Your Team
The most dangerous failures in multi-agent systems don't happen inside agents. They happen between them. Every agent can complete successfully and still produce a catastrophic outcome if the handoffs are opaque.
Start by instrumenting your handoff boundaries with the five elements above. Use OpenTelemetry's semantic conventions for GenAI agent spans as your baseline. Track context_keys_dropped and guardrails_inherited as explicit span attributes.
Set explicit context window limits on every handoff. Don't rely on silent truncation. Log a warning when truncation occurs and make that warning queryable.
If you're running evaluations at handoff boundaries, measure the latency. External API calls for faithfulness checks will become your bottleneck at production volume.
For regulated verticals, immutable handoff traces satisfy audit requirements. You need to demonstrate that every agent decision is traceable, that guardrail policies were enforced, and that context wasn't improperly disclosed. A complete handoff trace provides this evidence without manual reconstruction.
As you scale from three-agent pipelines to dozens of coordinated agents, the handoff surface area grows exponentially. The teams that invest in structured handoff tracing now will be the ones capable of running complex agentic systems at production volume. The teams that rely on flat logs will spend their time reconstructing failures that a single span attribute would have caught.
