Your fraud detection model shipped six months ago with stellar validation metrics. Today, false positives are climbing and you're scrambling to understand why. The likely culprit isn't model decay in the traditional sense, it's feature drift, the silent performance killer that happens when input distributions shift away from your training baseline.
This checklist helps your team implement systematic feature drift detection. You'll integrate feature platforms with AI observability tools to catch distribution shifts before they degrade predictions.
What This Checklist Covers
Feature drift occurs when the statistical properties of your model inputs change over time. Unlike prediction drift, which measures output distribution changes, feature drift gives you an early warning system. You catch problems upstream, before customers see degraded predictions.
This checklist assumes you're operating models in production and need structured monitoring. You should have:
- Models serving predictions in production environments
- Access to both training data and live inference data
- Authority to implement monitoring infrastructure
- A defined escalation path when drift is detected
Prerequisites
Before implementing feature drift detection, verify you have:
Training Baseline Access
You need the original training dataset with ground truth labels. Without this baseline, you can't measure drift. Good looks like: versioned training data stored in a queryable format with timestamps and event identifiers intact.
Feature Definition Consistency
Your offline training features and online serving features must use identical calculation logic. Good looks like: a single codebase defining features for both contexts, preventing training-serving skew from day one.
Event Tracking Infrastructure
You must log prediction events with unique identifiers, timestamps, feature values, and predictions. Good looks like: every inference event generates a structured log entry that ties predictions back to specific feature values.
Monitoring Platform Access
Your team needs read-write access to an AI observability platform capable of tracking distribution statistics. Good looks like: automated baseline comparisons running continuously, not manual CSV exports reviewed monthly.
Feature Drift Detection Checklist
1. Register Your Model with Baseline Training Data
Upload your training dataset to your monitoring platform. Include input features, predictions, and ground truth labels. Specify your model task type (classification, regression) and identify which columns represent events, timestamps, features, predictions, and targets.
Good looks like: Your monitoring platform displays training distribution histograms for each feature and prediction, establishing the reference point for all future comparisons.
2. Define Time-Consistent Feature Retrieval
Implement point-in-time correct feature lookups. When generating training data, retrieve feature values exactly as they existed at each event timestamp, not as they exist today.
Good looks like: A data scientist requesting features for a transaction from March 15 receives the 30-day moving average calculated on March 15, not today's 30-day average.
3. Establish Feature Logging in Production
Configure your serving infrastructure to log feature values, predictions, and event metadata for every inference. Route these logs to your monitoring platform.
Good looks like: Within minutes of serving a prediction, your monitoring platform receives the event ID, timestamp, all input feature values, and the prediction. You don't wait for batch uploads.
4. Configure Drift Detection Metrics
Set up distribution comparison metrics for each feature. Common choices include Population Stability Index (PSI), Kullback-Leibler divergence, or Jensen-Shannon distance. Define comparison windows (daily, weekly) and baseline periods.
Good looks like: Your platform automatically calculates PSI scores comparing this week's feature distributions to training baselines, flagging scores above 0.1 as moderate drift and above 0.25 as severe drift.
5. Set Drift Alerting Thresholds
Define what constitutes actionable drift for your use case. Not every distribution shift requires immediate action. Consider business impact, regulatory requirements, and team capacity.
Good looks like: Alerts fire when two or more features show PSI > 0.25 for three consecutive days, or when any single feature exceeds PSI > 0.5. Alerts route to a monitored channel with clear escalation procedures.
6. Link Ground Truth Labels to Events
Implement delayed ground truth logging. When actual outcomes become known (fraud confirmed, loan repaid, diagnosis verified), upload them to your monitoring platform using the original event identifiers.
Good looks like: Fraud labels arrive 30-90 days after transactions. Your monitoring platform joins these labels to logged predictions using transaction IDs, enabling performance metric calculation alongside drift metrics.
7. Monitor Prediction Drift Alongside Feature Drift
Track changes in your model's output distribution. Prediction drift often lags feature drift but indicates your model is responding to input changes.
Good looks like: Your dashboard shows feature drift charts above prediction drift charts. You can visually correlate upstream distribution shifts with downstream prediction changes.
8. Document Drift Investigation Procedures
Write runbooks specifying who investigates drift alerts, what analysis they perform, and what actions they can take. Include decision trees for retraining, recalibration, or model retirement.
Good looks like: When alerted, your model risk analyst knows to check upstream data pipeline changes first, then seasonal patterns, then structural market shifts. They have authority to trigger retraining or flag the model for validation review.
9. Schedule Regular Drift Reviews
Establish a cadence for reviewing drift metrics even when no alerts fire. Gradual drift below alert thresholds can accumulate into meaningful performance degradation.
Good looks like: Monthly model review meetings include drift trend charts for all production models. Teams discuss borderline cases and adjust thresholds based on observed patterns.
10. Version Your Baselines
When you retrain a model, create a new baseline from the updated training data. Maintain historical baselines to understand how your models and data have evolved.
Good looks like: Your monitoring platform stores baseline versions tagged with model version numbers and training dates. You can compare current production data to any historical baseline.
Common Mistakes
Monitoring Predictions Without Features
Teams often track prediction drift but ignore feature drift. You're treating symptoms, not causes. By the time prediction drift appears, your model has already been serving degraded outputs for days or weeks.
Using Current Feature Values for Training Data
Retrieving today's feature values for historical events guarantees training-serving skew. Your model trains on data it could never have seen in production.
Setting Uniform Thresholds Across Features
Not all features matter equally. A 20% shift in a low-importance feature might be irrelevant while a 5% shift in your most predictive feature could be critical. Weight your alerting by feature importance scores from your model.
Ignoring Seasonal Patterns
December retail transaction patterns differ from July patterns. That's not drift; that's seasonality. Use rolling baselines or seasonal adjustments to avoid alert fatigue.
Logging Only Aggregates
Summary statistics hide distribution changes. Log individual event-level feature values, not just daily averages. You need the full distribution to calculate proper drift metrics.
Next Steps
Start with your highest-risk production model. Implement items 1-3 this sprint to establish logging infrastructure. Add drift detection (items 4-7) next sprint. Operationalize with procedures and reviews (items 8-10) within 90 days.
Your monitoring platform should track 30+ out-of-the-box ML metrics covering drift, performance, and data quality. If you're manually calculating drift scores in spreadsheets, you're not monitoring, you're auditing after the fact.
Feature drift detection isn't optional for regulated models or high-stakes predictions. SR 11-7 requires ongoing monitoring and performance testing. The EU AI Act mandates post-market monitoring for high-risk systems. Feature drift detection gives you the early warning system to meet these obligations before performance degradation becomes a compliance finding.
