An LLM Evaluation Checklist You Can Run Today

Your LLM-powered system passed every benchmark. Then it shipped, and within a week it started citing policies that don't exist. The model scored well; the system failed.

You need an evaluation framework that measures what matters: how your application behaves on your tasks, your data, and your policies. This checklist gives you the structure to build one.

Purpose of This Checklist

This is a production-ready evaluation framework for LLM-powered applications. It covers five stages: defining what you're measuring, building datasets that represent real use, choosing scorers that fit each task, running pre-release validation, and extending those same evaluators into production monitoring.

The framework applies to retrieval-augmented generation systems, multi-step agents, guardrails, and full workflows. It's built for teams that need to prove system behavior before and after launch.

Prerequisites

Before you start:

• Access to representative production data. You need real inputs your system will encounter. Synthetic data won't expose the failure modes that matter.
• Frozen golden set. Reserve a subset of representative data that stays unchanged across releases. This lets you compare versions cleanly.
• Defined quality criteria. Know what "good" means for your use case. Is it factual grounding? Policy compliance? Appropriate tone? Write it down before you measure.
• Evaluation environment. You'll need infrastructure to run scorers at scale, both offline and in production.

If you're evaluating systems that handle regulated data, confirm your evaluation environment meets the same data residency and access controls as production.

The Evaluation Checklist

1. Define Your Evaluation Target

Specify exactly what you're scoring. Don't evaluate "the model" when you mean "the application."

• RAG pipeline: Retrieval quality, generation quality, groundedness
• Multi-step agent: Each tool call, decision path, final output
• Single guardrail: Input filter or output validator in isolation
• Full workflow: End-to-end user interaction across multiple turns

Write a one-sentence description of what passes and what fails. If you can't articulate it clearly, your evaluators won't measure it consistently.

2. Build Representative Datasets

Assemble inputs that mirror real traffic distribution. Don't oversample edge cases or cherry-pick easy examples.

• Golden set (frozen): 500-2,000 examples covering core use cases. Lock this dataset and never modify it. Every release gets scored against the same baseline.
• Regression set (growing): Start with 200 examples. Add every production failure you find. This set evolves as your system learns new failure modes.
• Adversarial set (optional): 100-300 examples designed to test boundaries: prompt injections, policy violations, ambiguous instructions.

For each example, record:

• Input (user query or system prompt)
• Expected behavior (reference answer, required grounding, prohibited content)
• Metadata (source, difficulty, risk tier)

3. Choose Scorers Per Target

Match your scorer to the task. Structured extraction rewards deterministic checks; open-ended answers need semantic scoring or an LLM judge.

For Structured Output

• Exact match: JSON schema validation, required field presence
• Format validators: Length constraints, forbidden patterns, entity extraction accuracy

For Semantic Quality

• Embedding-based similarity: BERTScore for meaning overlap when reference answers exist
• Natural language inference: Check if claims are entailed by source documents

For Open-Ended Quality

• LLM-as-a-Judge with rubric: Score faithfulness, relevance, completeness, tone
• Pairwise comparison: Rank two model versions to decide which ships

Write explicit rubrics for LLM judges. Vague criteria like "answer quality" produce inconsistent scores. Specific criteria like "all claims must cite a source document by title and section" produce repeatable verdicts.

# Sample rubric for a policy-grounded assistant
faithfulness:
  description: "Every claim is supported by a cited policy document"
  scale: 1-5
  5: "All claims cite specific policy sections"
  3: "Most claims grounded, some unsupported"
  1: "Contains fabricated policy details"

relevance:
  description: "Answer addresses the question asked"
  scale: 1-5
  5: "Directly answers the question with no tangents"
  3: "Partially relevant with some off-topic content"
  1: "Does not address the question"

completeness:
  description: "No required detail is missing"
  scale: 1-5
  5: "All required steps or conditions explained"
  3: "Key details present but some gaps"
  1: "Critical information omitted"

4. Run Offline Evaluation Before Release

Use your golden set for model selection and regression detection. No version ships if it degrades scores.

• Baseline the current system: Run all scorers on the golden set. Record scores per example and aggregate metrics.
• Compare candidate versions: Score each candidate on the same golden set. Look for regressions on high-value examples.
• Set pass thresholds: Define minimum scores per criterion. A 20% drop in faithfulness is a release blocker.
• Document failures: When a candidate fails, record the input, the output, and the scorer verdict. Feed these into your regression set.

Track scores over time. If your baseline faithfulness score is 4.2 and a new version scores 3.8, you've found a regression before users did.

5. Extend Evaluators Into Production Monitoring

The same evaluators that scored your golden set now score live traffic. This keeps measurement consistent across the release boundary.

• Online scoring: Run evaluators on a sample of production traces. Start with 1-10% of traffic.
• Canary releases: Deploy new versions to a small user segment first. Compare live scores to the baseline before full rollout.
• Drift detection: Track score distributions weekly. A sudden drop in relevance scores signals a data shift or upstream failure.
• Feedback loop: When production monitoring flags a failure, add that example to your regression set. Re-run offline evaluation to confirm the fix.

If you're using an external LLM as a judge over an API, cost scales with traffic. Scoring 100,000 traces per day means 100,000 API calls. Consider in-environment evaluators that return verdicts in under 100ms with no external dependency.

Customizing This Checklist

Adapt the framework to your risk profile:

• High-stakes systems (financial services, healthcare): Increase golden set size to 2,000+ examples. Add human review for 10% of LLM judge verdicts to validate scorer alignment.
• Rapid iteration environments: Keep the golden set smaller (500 examples) but refresh the regression set weekly as you find new failure modes.
• Multi-tenant systems: Build separate golden sets per tenant or domain. A legal assistant and a customer support bot need different evaluation criteria.

Validation Steps

Before you trust this framework in production:

1. Validate judge agreement. Run your LLM-as-a-Judge scorer on 100 examples. Have a human reviewer score the same examples. If agreement is below 80%, refine your rubric.
2. Confirm golden set coverage. Does your frozen set represent the input distribution you see in production? If 30% of live traffic is edge cases but your golden set is all common queries, you're measuring the wrong thing.
3. Test the feedback loop. Manually inject a known failure into production monitoring. Confirm it gets flagged, added to the regression set, and caught by offline evaluation on the next release.
4. Model the cost. If you're using external LLM judges, calculate the monthly cost at your expected traffic volume. If it's prohibitive, explore in-environment evaluators or down-sample strategically.

The strongest signal that your evaluation framework works: you catch a regression in offline evaluation that would have caused a production incident. When that happens, you've built something real.