What This Checklist Is For
Your model validation process likely catches issues like miscalibrated predictions, overfitting, and data leakage. But does it identify models that pass validation by being so vague they can't be wrong?
This checklist helps you spot models with overly broad confidence intervals or prediction ranges that technically pass validation metrics but offer no decision value. Imagine a credit risk model predicting default probability between 2% and 95% for every applicant, or a fraud detector flagging "possible fraud" on 80% of transactions. These models can't be proven wrong because they never commit to a meaningful prediction.
Use this checklist during independent model validation reviews, quarterly monitoring assessments, or when stakeholders complain that "the model isn't helpful" despite passing all technical tests.
Prerequisites
Before using this checklist, ensure you have:
- Access to model outputs: Raw predictions, confidence intervals, probability ranges, or classification scores.
- Historical performance data: At least one full validation cycle's worth of predictions and outcomes.
- Business decision context: Understanding of how predictions feed into actual decisions (approval thresholds, alert workflows, capital allocation).
- Baseline comparison: A simple heuristic, prior model version, or industry benchmark for comparison.
You don't need advanced statistical tools. Most checks work with basic percentile calculations and frequency counts in your existing validation toolkit.
The Checklist
Section 1: Interval Width Analysis
Purpose: Determine how often your model's predictions are too broad to inform decisions.
Calculate median interval width across all predictions in your validation set. For regression models, this is the difference between upper and lower confidence bounds. For classifiers, measure the range between minimum and maximum class probabilities.
Compare interval width to decision threshold spread. If your business uses a 60% probability threshold for "high risk," and 70% of predictions fall between 40% and 80%, those intervals span your decision boundary and force human judgment anyway.
Measure the percentage of predictions where interval width exceeds 50% of the possible range. A model predicting "between 15% and 85%" on a 0-100% scale gives a 70-point range. Flag any model where more than 20% of predictions show this pattern.
Track interval width over time. Models that start precise and gradually widen their intervals are often detecting their own performance degradation and compensating by hedging.
Section 2: Decision Impact Assessment
Purpose: Test whether the model's output actually changes decisions compared to doing nothing.
Run a null model comparison. Build the simplest possible alternative (population base rate, random assignment within observed ranges, previous period's value). Calculate how often your model's recommended action differs from the null model's action.
Identify the "uncertainty zone" where your model refuses to commit. For binary classifiers, this is typically predictions between 0.3 and 0.7. Measure what percentage of your production volume falls here.
Calculate decision-adjusted accuracy. Don't just measure whether predictions are correct; measure whether they're useful. If a fraud model scores 60% of transactions as "moderate risk" and you investigate all of them anyway, the model isn't reducing workload.
Document override rates. If business users override or ignore model outputs more than 30% of the time, your intervals are probably too wide to be actionable.
Section 3: Calibration Specificity
Purpose: Verify the model commits to predictions across the full range, not just the safe middle.
Plot prediction distribution. Your model should produce predictions across its full range. If you're building a probability model and 90% of predictions fall between 0.4 and 0.6, you have an interval width problem disguised as a calibration problem.
Test boundary cases separately. Validate performance on the tails (predictions below 10th percentile and above 90th percentile). Models that are accurate in the middle but vague at the extremes often widen intervals to avoid being wrong on edge cases.
Check for artificial floor/ceiling effects. Some models learn to never predict below 5% or above 95% because the training process penalized extreme predictions that turned out wrong. This creates a built-in 90-point interval on every prediction.
Section 4: Comparative Benchmarking
Purpose: Establish whether your model's intervals are justified by genuine uncertainty or defensive hedging.
Compare to simpler models on the same data. If a logistic regression produces 30-point average intervals and your neural network produces 60-point intervals, the complexity isn't buying you precision.
Benchmark against disclosed model performance. For vendor models or outsourced models, request interval width statistics from their validation evidence. Your model should perform comparably on your data.
Test interval width by subpopulation. Models that produce tight intervals on easy cases and wide intervals on hard cases are working as designed. Models that produce uniformly wide intervals everywhere are hedging.
How to Customize It
For regression models: Replace "confidence interval" language with "prediction interval" and adjust the 50% threshold based on your outcome variable's range. If you're predicting dollar losses, a $50,000 interval on a $100,000 expected value is different than a $50,000 interval on a $1M expected value.
For classification models: Focus on Section 2 (Decision Impact) and Section 3 (Calibration Specificity). Add a check for "predicted probability standard deviation" if your model outputs class probabilities rather than hard intervals.
For time-series models: Add temporal checks. Measure whether interval width increases with forecast horizon, and whether it increases faster than theoretical uncertainty should grow.
For high-stakes use cases (credit decisions, medical diagnosis, safety systems): Tighten the thresholds. Flag models where more than 10% of predictions span the decision boundary, and require documented justification for any interval wider than 30% of the possible range.
Validation Steps
After completing the checklist, validate your findings:
Present interval width statistics to model stakeholders. Ask: "If the model tells you the risk is between 20% and 80%, what decision do you make?" If the answer is "we investigate manually," the model isn't adding value.
Calculate cost of uncertainty. Estimate how much your organization spends on manual review, secondary screening, or conservative decisions because model outputs are too vague to act on.
Document in your validation report. SR 11-7 requires assessment of model limitations and use restrictions. Overly broad intervals are a limitation that should trigger use restrictions (for example: "model outputs should not be used as sole decision input when predicted interval exceeds X").
Set ongoing monitoring thresholds. Add "percentage of predictions with interval width >50%" to your quarterly model monitoring dashboard. Treat widening intervals as a leading indicator of model degradation.
If more than 30% of your model's predictions fail these checks, you don't have a validation problem. You have a model that's learned the safest way to pass validation is to never be specific enough to be wrong.
