Your LLM is live, traffic is climbing, and you're staring at a monitoring bill that's about to hit six figures because you're evaluating every single request.
This is the operational question every Model Risk & Assurance team faces once continuous monitoring moves from concept to production: do you evaluate 100% of your LLM traffic, or do you sample? The answer depends on three factors: your risk tolerance, your traffic volume, and whether you're using external APIs or in-environment models.
The Decision You're Facing
You need to detect hallucinations, safety violations, and performance degradation in production. The question isn't whether to monitor, it's how much coverage you need and what infrastructure approach makes that sustainable.
This isn't a one-time architecture choice. Your monitoring strategy should evolve as your deployment matures. What works at 10,000 queries per day breaks at 500,000. What's acceptable for an internal tool isn't sufficient for a customer-facing application.
Key Factors That Affect Your Choice
Traffic volume determines your baseline cost. At 500,000 traces per day using external API evaluation, you're looking at approximately $260K annually. That's the AI Trust Tax: the per-query cost of calling a third-party LLM to judge your outputs. Your actual cost varies by model choice, deployment size, and traffic patterns, but the relationship is linear. Double your traffic, double your bill.
Risk tolerance sets your coverage floor. High-risk applications in regulated environments can't tolerate sampling strategies that might miss critical failures. If you're deploying an LLM for medical advice, loan decisions, or legal guidance, you need near-complete coverage with immediate alerting on policy violations.
Infrastructure approach changes the economics. In-environment evaluation models eliminate the per-query API cost. You pay for compute capacity instead of per-request fees, which shifts the cost curve dramatically at scale. The break-even point typically falls between 50,000 and 100,000 queries per day, depending on your cloud pricing and evaluation model requirements.
Evaluation speed affects user experience. If you're running synchronous evaluation (blocking the response until scoring completes), every millisecond matters. Asynchronous evaluation decouples scoring from serving, letting you run more comprehensive checks without impacting Time to First Token.
Path A: Full-Traffic Evaluation with In-Environment Models
Choose this path when:
- You're operating a high-risk application where missing a single failure isn't acceptable
- Your traffic volume makes per-query API costs prohibitive (generally above 100K queries/day)
- You need real-time alerting on safety violations or hallucinations
- Data residency requirements prevent sending outputs to external APIs
- You can provision dedicated compute for evaluation workloads
This approach requires upfront infrastructure investment but scales predictably. You're running purpose-built evaluation models for specific tasks: faithfulness scoring, safety violation detection, and jailbreak attempt identification. These models execute inside your environment without external API calls.
Your operational metrics matter here. Track Time to First Token separately from evaluation latency. If evaluation is synchronous, you're adding that latency to every user request. Most teams run evaluation asynchronously: serve the response immediately, score it in parallel, and alert on failures.
Set up tiered alerting. Not every low faithfulness score requires immediate escalation, but certain patterns do: sustained degradation over an hour, sudden spikes in hallucination rates, or any detection of prohibited content. Your alerting logic should distinguish between isolated anomalies and systematic failures.
Path B: Intelligent Sampling with External APIs
Choose this path when:
- You're in early deployment with traffic below 50K queries per day
- You need evaluation flexibility without infrastructure overhead
- Your application isn't classified as high-risk under regulatory frameworks
- You can tolerate delayed detection of emerging issues (hours, not minutes)
- Your evaluation needs change frequently and you want to experiment with different judge models
Sampling strategies range from simple random selection to sophisticated risk-based approaches. Start with stratified sampling: ensure you're capturing examples from each user segment, prompt category, and time window. A pure random sample might miss rare but critical failure modes.
Risk-based sampling concentrates evaluation resources where failures matter most. Score every query using fast, cheap heuristics (response length, uncertainty signals, keyword triggers), then run expensive LLM-as-a-Judge evaluation only on high-risk candidates. This hybrid approach catches more issues per evaluation dollar spent.
Your sampling rate determines detection latency. At 10% sampling, you need 10x the traffic before you'll catch a 1% failure rate with statistical confidence. If you're seeing 10,000 queries per day and sampling at 10%, you're evaluating 1,000 queries. A failure mode affecting 50 queries per day (0.5%) might not appear in your sample for days.
Track your coverage explicitly. Don't just measure "percentage of traffic sampled," measure "time to detection for known failure modes." Run synthetic canary queries with intentional issues to verify your sampling strategy would catch them within your acceptable detection window.
Path C: Hybrid Coverage with Tiered Evaluation
Choose this path when:
- You have mixed-risk workloads in a single deployment
- Traffic volume is high but budget for full evaluation isn't approved
- You need immediate detection on critical failures but can tolerate delayed detection on quality issues
- Your application serves both internal and external users with different risk profiles
This approach runs different evaluation strategies for different query types. Evaluate 100% of customer-facing queries using lightweight in-environment models for safety and hallucination detection. Sample 10-20% of internal queries for quality metrics using external API judges. Route queries flagged by fast heuristics through comprehensive multi-metric evaluation.
Your infrastructure needs a routing layer that classifies queries by risk tier before evaluation. This classification happens at request time based on user type, prompt content, and application context. High-risk queries go through full evaluation pipelines. Low-risk queries get sampled or evaluated with faster, cheaper methods.
The challenge is maintaining this classification logic as your application evolves. Every new feature needs a risk assessment that determines its evaluation tier. Your governance documentation should specify which query types require full coverage and which allow sampling, with clear rationale tied to SR 11-7 risk tiering or your internal model risk framework.
Summary Matrix
| Approach | Best For | Coverage | Cost at 500K/day | Detection Speed |
|---|---|---|---|---|
| Full traffic, in-environment | High-risk applications, regulated environments | 100% | Infrastructure fixed cost | Real-time |
| Sampled, external API | Early-stage deployments, low-risk applications | 5-20% | $13K-$52K annually | Hours to days |
| Hybrid tiered | Mixed-risk workloads, budget constraints | 100% critical, 10-20% other | Infrastructure + $10K-$30K | Real-time for critical paths |
Your monitoring strategy isn't static. Start with the approach that matches your current scale and risk profile, then migrate as both evolve. Most teams begin with sampled external evaluation during initial deployment, shift to hybrid coverage as traffic grows, and move to full in-environment evaluation once they hit the cost break-even point or face regulatory requirements that demand complete coverage.
The wrong choice isn't sampling when you should evaluate everything. The wrong choice is not knowing which queries you're missing and how long it would take to detect a systematic failure in production.
