Most teams evaluating AI observability tools approach coding agents with the same checklist they used for chatbots. They ask about framework support, trace volume, and dashboard quality. Then they deploy, only to discover that their observability stack captures everything except what matters: file mutations, shell commands, and recursive retry loops that turn a single flawed reasoning step into a production incident.
These aren't configuration errors. They're structural blind spots that emerge when you apply LLM observability patterns to agents that modify codebases. Here's what goes wrong, why it keeps happening, and how to fix it before your next deployment.
Why These Mistakes Keep Happening
Coding agents have crossed a capability threshold that most observability tools weren't designed for. They don't just generate text; they execute commands, modify files, and trigger downstream actions with compounding effects. The observability frameworks you're using were built for applications that return responses, not agents that alter infrastructure.
The procurement process exacerbates this issue. Vendors demo trace collection and evaluation dashboards. Teams check boxes on a feature matrix. No one asks whether the tool can enforce policy before an unsafe action executes, because that question doesn't appear on the standard RFP template. By the time the gap becomes obvious, the agent is already in production.
Mistake 1: Sampling Traces During Development
Why it happens: Teams enable trace sampling early to control storage costs. It's common for high-volume applications, and most observability vendors recommend it.
The consequence: Coding agents fail in specific, high-complexity sessions: the ones where reasoning chains branch deeply, retry loops compound, or file mutations cascade across dozens of targets. Those sessions generate the most spans, which makes them the first ones your sampling rate will drop. When you need to reconstruct what went wrong, the trace is incomplete.
The fix: Capture 100% of traces during development and early production. Don't sample until you've established baseline failure patterns and confirmed that your evaluation logic catches them. For coding agents, the sessions that matter most are the ones that generate the most trace data. Storage costs during this phase are minor compared to the cost of debugging a production incident without complete lineage.
Mistake 2: Evaluating Only the Final Output
Why it happens: Most teams treat evaluation like code review: check the final commit, run the tests, approve or reject. That pattern works for human-authored code. It fails for agents.
The consequence: A coding agent can produce syntactically correct code through a path that violated your security policies, exceeded its tool-call authority, or modified files it shouldn't have touched. If you evaluate only the final output, you miss the upstream violations. You also miss cost-inefficient retry loops that burned through your token budget before arriving at a working solution.
One documented incident involved a coding agent trapped in a failing test retry loop that triggered roughly 800 automated backend requests in minutes, draining Pro credit allocations entirely. The final code passed tests. The evaluation logic never flagged the cost spiral that preceded it.
The fix: Implement per-span evaluation that runs at every reasoning step, not just the final output. For coding agents, that means checking file-level attribution (which files were accessed and why), command execution logs (what shell commands ran and what they returned), and policy compliance at each tool call. Assertion-based testing catches the specific failure modes that matter: unauthorized repository access, deployment commands that bypass change management, and retry loops that exceed cost thresholds.
Mistake 3: Treating Tracing as Observability
Why it happens: Tracing is the most visible layer of observability. It's what vendors demo first. Teams assume that if they can see what the agent did, they've solved the observability problem.
The consequence: Tracing shows what happened. It doesn't tell you whether it was correct, and it doesn't stop anything. Without evaluation and enforcement, you're left reviewing a record of actions that already executed, some of which should have been blocked before they ran. Cascading file mutations, tool-call authority creep, and unauthorized dependency installations all show up clearly in your traces, after the damage is done.
The fix: Structure your observability stack in three layers: execution tracing (what happened), in-environment evaluation (whether it was correct), and enforceable policy (what gets stopped before it runs). Tracing is foundational, but it's not sufficient. For compliance requirements under SOC 2 or the EU AI Act, you need decision lineage: traceable reasoning tied to specific policies, auditable by humans when questions arise. That requires governance tooling, not just trace collection.
Mistake 4: Applying Static Guardrail Policies
Why it happens: Teams define a single set of guardrails and apply them uniformly across all repositories and environments. It's the simplest deployment model, and it mirrors how most security policies are implemented.
The consequence: Coding agents operate across repositories with different permission models. A guardrail policy strict enough for your production monorepo will block legitimate actions in experimental branches. A policy permissive enough for rapid prototyping will allow risky actions in restricted environments. Static policies either over-block or under-protect; there's no middle ground.
The fix: Scope guardrail policies to repository and environment context. Your production repositories should enforce code review requirements before any agent commits changes. Your staging environments can permit broader tool-call authority. Your experimental branches can allow file mutations that would be blocked elsewhere. Most AI observability platforms don't support context-scoped policies natively; if yours doesn't, you'll need to implement policy routing at the orchestration layer before agent actions reach your codebase.
Mistake 5: Ignoring Evaluation Economics at Scale
Why it happens: During proof-of-concept phases, per-evaluation API costs are invisible. Teams focus on accuracy metrics and dashboard usability. No one models what evaluation will cost at production trace volumes.
The consequence: Many AI observability platforms rely on external LLM-as-a-Judge calls for every span. At prototype scale, that's manageable. At production scale, those per-call API costs compound with trace volume. Consider a team running 50,000 evaluated traces per week, each generating 20 spans. If each evaluation call costs $0.002, you're adding $2,000 per week in evaluation costs alone, separate from your observability platform subscription.
The fix: Ask vendors explicitly how evaluation runs: external API calls or in-environment models. If the platform depends on external calls, model your evaluation costs at your projected trace volume before signing. In-environment evaluation eliminates per-call costs entirely but requires the vendor to maintain evaluation models as part of the platform. Fiddler Centor Models, for example, run evaluation in-environment with no external API dependency. Evaluate vendors on evaluation economics, not just evaluation accuracy.
Prevention Checklist
Before deploying coding agents with your current observability stack:
- Confirm 100% trace capture is enabled for development and early production environments
- Verify per-span evaluation runs at every reasoning step, not just final output
- Test that file-level attribution captures which files were read, modified, or created
- Validate that command execution logs show exactly what shell commands ran
- Implement guardrail policies scoped to repository and environment context
- Model evaluation costs at projected production trace volumes
- Verify OpenTelemetry compatibility to prevent vendor lock-in on trace data
- Confirm governance tooling creates decision lineage tied to specific policies
- Test policy enforcement blocks unauthorized actions before they execute
- Document trace data ownership and export capabilities
The coding agents your team deploys next month will be more capable than the ones running today. Your observability stack needs to be ready before they are.
