Enterprise AI systems are consuming token budgets faster than finance teams can track. Recent data reveals a cost structure that standard dashboards miss, and the gap between pilot economics and production reality is wider than most governance teams anticipated.
What the Data Shows
Token costs in production AI systems follow a three-part structure that compounds as systems scale. Teams often waste 40 to 60 percent of their token budgets on inference inefficiency alone. A proof-of-concept costing $50 in API calls can scale to $2.5 million per month at production volume. In one deployment, total monthly AI operating costs rose from $1,500 during POC to just over $1 million in production, a 717x increase. The LLM API line item within that same deployment jumped from $500 to $847,293, roughly 1,700x on its own.
These aren't outliers. Enterprise LLM API spending hit $8.4 billion in the first half of 2025. Microsoft reportedly canceled internal Claude Code licenses after employee usage scaled beyond expectations. GitHub shifted Copilot to usage-based billing tied directly to token consumption. The pattern is consistent: per-token prices fall, but aggregate costs rise because agentic workloads consume far more tokens per task.
Five Key Findings
1. Evaluation costs scale independently from inference costs
Every AI output requires evaluation. When evaluation runs against external LLMs, it generates a separate per-call charge that behaves differently from inference spend. This cost is driven by trace volume and evaluation count, not by model choice or context size. At 100,000 traces per day with three evaluation metrics per trace, you're adding 300,000 external LLM calls daily to your bill. At high volume, evaluation can cost more than the inference it monitors. Most teams lump all LLM API charges into a single line item, masking evaluation overhead entirely.
2. Down-sampling creates incident exposure that dwarfs token savings
When evaluation costs become unsustainable, teams down-sample, evaluating 10 percent of traces and hoping the other 90 percent behave. Those unmonitored traces become your incident surface. A single undetected hallucination in a financial advisory agent creates regulatory exposure that no amount of token optimization will offset. The incidents that matter most are often edge cases hiding in the unsampled majority.
3. Multi-agent systems accelerate costs non-linearly
Each agent call triggers sub-agent calls, tool invocations, and context window expansion. A single user request to an orchestration agent might fan out to five specialized agents, each consuming its own token budget across input, reasoning, and output. A single user interaction can consume tens of thousands of tokens across the agentic hierarchy. Agents pulling data through MCP servers and tool endpoints consume tokens on retrieval, not just generation.
4. Context window inflation wastes more than model selection
Poor data serialization and over-retrieval routinely consume a significant share of token budgets. A retrieval step that returns 20 document chunks when 3 are relevant wastes tokens on context the model must process but gains nothing from. Using a frontier reasoning model to make a binary classification, stuffing 80,000 tokens of context into every call when 5,000 would do, or running identical prompts thousands of times an hour with no caching, each pattern consumes tokens that produce no incremental value.
5. Shadow token costs hide in adjacent systems
Tokens consumed by evaluation, guardrail checks, and observability tooling never show up in the primary application's billing view, but they hit the same LLM provider bill. Teams optimizing their application's token consumption often overlook these adjacent costs entirely. When evaluation runs against external LLMs, the evaluation bill can approach or exceed the inference bill at high trace volumes.
What This Means for Your Team
You can't govern what you can't see. Aggregate daily spend dashboards hide the attribution data you need to act on. A daily total of $15,000 in token spend isn't actionable. Knowing that your document-summarization agent accounts for 60 percent of that spend, and that 80 percent of its tokens go to context window stuffing, is.
The compounding effect is what catches teams off guard. As agent fleets grow, the cost surface expands across all three causes at once: more inference tokens, more traces to evaluate, and a larger unmonitored surface when teams inevitably down-sample. Cutting inference waste without addressing evaluation overhead means you still carry a linear cost curve on every trace. Eliminating evaluation costs by down-sampling expands your incident exposure.
Token governance is emerging as a structural requirement, not an operational optimization. The fix starts with instrumentation and attribution, not with switching to cheaper models.
Action Items by Priority
Priority 1: Instrument every call with cost attribution
Capture model identifier, input and output token counts, the feature or agent that triggered the call, latency, and estimated cost per request. Tag every call with the originating agent and distinguish inference from evaluation. Without per-call instrumentation, cost attribution is guesswork.
Priority 2: Separate inference costs from evaluation costs in your telemetry
Separating them requires tagging evaluation calls at the telemetry layer. This distinction reveals whether your evaluation overhead is approaching or exceeding your inference costs, a common pattern at high trace volumes.
Priority 3: Implement per-agent and per-span attribution
Per-agent attribution reveals which parts of the system burn the most. Build this into your observability layer from day one, before production scale makes retroactive instrumentation prohibitively expensive.
Priority 4: Set alerts on token spend thresholds
A cascading agent loop can burn through a week's budget in hours if nothing triggers an alert. Monitor for cost anomalies so spikes are caught before they compound.
Priority 5: Address context window inflation in RAG pipelines
Audit your retrieval step for over-retrieval. Review your data serialization formats for verbosity. These patterns are easier to fix than multi-agent orchestration logic, and they often deliver the fastest ROI.
