The Conventional Wisdom
Your finance team likely treats API costs as a single line item. You budget for inference, track monthly spend, and calculate ROI by dividing business value by total API cost. If your document processing agent saves $50K in labor but costs $30K in API calls, you report a 67% ROI. Simple, but misleading.
This approach assumes all API costs serve the same function. A dollar spent on inference and a dollar spent on evaluation both appear on your LLM provider bill, so they both go into the denominator. This practice is common because it's how the bills arrive: one provider, one monthly charge, one number for finance to track.
Why This View Is Incomplete
Evaluation costs aren't just inference costs under a different name. They're a separate expense category that scales independently, serves a different function, and requires different mitigation strategies. Combining them creates a distortion that makes high-performing agents seem like marginal investments.
Take a financial services team running loan application reviews. Inference costs might be budgeted at $15,000 per month. However, every agent output triggers separate API calls for faithfulness scoring, groundedness checks, and toxicity screening. At 100K agent actions per month, evaluation calls can match or exceed inference spend. At 1M actions, evaluation becomes the dominant line item.
The agent performs well, and the business impact is real. But the ROI calculation shows a lower number than the labor savings suggest because the denominator includes costs unrelated to the agent's core function. You're measuring the cost of quality assurance as if it were the cost of production.
This isn't just a budgeting oversight; it's a category error. Evaluation costs break into three components, each invisible in standard ROI frameworks:
The Evaluation Trust Tax is the per-call cost incurred when external LLMs score your outputs. Three metrics across 500K monthly traces generate 1.5M evaluation API calls per month. These charges appear on the same bill as inference, making them effectively invisible until cumulative costs are significantly higher than projected. A 2024 Gartner survey found that fewer than half of AI projects (48%) make it from prototype into production, partly due to operational cost and complexity.
Incident Risk Exposure occurs when evaluation costs force down-sampling. Instead of evaluating every trace, you evaluate 1% to 5%. The remaining 95% to 99% run without quality assurance. Hallucinations pass through, policy violations go unrecorded, and data exposure events are never flagged. The IBM Cost of a Data Breach Report found the global average cost of a data breach is $4.4M, with AI-related incidents carrying additional risk when governance controls are absent. This exposure is absent from every standard ROI framework.
Operational Overhead affects teams that build custom evaluation infrastructure to avoid API costs. You trade variable costs for engineering costs. Building, maintaining, and iterating on in-house evaluation pipelines requires dedicated platform engineering effort. These hours rarely appear in ROI calculations because they're charged to platform budgets, not AI project budgets.
The Evidence
The math is straightforward. If you're running three evaluation metrics per trace, you're making three separate API calls to an external model for every agent action. Double your agent volume, double your evaluation bill. The cost relationship is linear.
Teams discover this three to six months into production when they ask why a high-performing agent is returning far less than its labor savings would suggest. The answer is in the denominator. You're dividing business value by a cost pool that includes quality assurance infrastructure, not just the agent's operational cost.
Four common miscounting patterns compound the problem:
You mistake token cost for total cost, optimizing prompt length while ignoring evaluation calls made on separate models. Reducing inference tokens doesn't reduce evaluation spend.
You down-sample without risk modeling, choosing a 1% sample rate based on budget constraints without quantifying incident exposure. A 1% sample means 99% of traces have zero quality assurance.
You evaluate with the generating model, using GPT-4o to score GPT-4o outputs. This introduces self-enhancement bias, where the evaluator favors its own output patterns. The evaluator should be independent of the generator.
You miss cost attribution across agents. When multiple agents share an evaluation pipeline, you can't determine which agent is driving evaluation spend without per-agent cost tagging.
What to Do Instead
Separate evaluation costs from inference costs in your accounting. Create distinct line items. Track them independently. Calculate ROI using inference costs only, then account for evaluation as quality assurance overhead, similar to unit testing or security scanning.
The structural solution is to move evaluation out of the variable-cost column entirely. In-environment evaluation means running evaluation models inside your own infrastructure. No external API call is made. No per-evaluation cost is incurred. No data leaves the environment.
With the per-call cost removed, the calculus changes. You can evaluate 100% of traces without the linear cost scaling that forces down-sampling. This eliminates both the Trust Tax and Incident Risk Exposure simultaneously. The ROI equation simplifies: evaluation becomes a fixed infrastructure cost tied to compute, not a variable API cost tied to trace volume.
Fixed costs don't scale with agent volume. With evaluation no longer scaling per trace, ROI stops being dragged down by a cost that grows with every request. Evaluation doesn't disappear (in-environment models still carry a fixed compute cost), but it becomes a predictable line item instead of a variable one that balloons with volume. Finance and engineering see the same number.
When the Conventional Wisdom Is Right
Single-line API accounting works when evaluation costs are negligible relative to inference. If you're running a low-volume prototype with minimal quality assurance requirements, the distortion is small enough to ignore. If your evaluation needs are met by rule-based checks that don't require external LLM calls, you're not carrying the Trust Tax.
The conventional view also holds when you're comparing agents on a relative basis rather than calculating absolute ROI. If all your agents carry similar evaluation overhead, the ranking stays valid even if the absolute numbers are distorted.
But once you're running production-scale deployments with comprehensive quality assurance, the category error becomes significant. You're systematically undervaluing your AI investments, and the dollars lost to the Trust Tax grow with every additional trace. The organizations that fix their cost accounting first will be the ones that can justify scaling their agent deployments next.
