Your model inventory might look orderly on paper, but can you articulate what happens when your production system encounters data it wasn't trained to handle? Or when it optimizes for a proxy metric that diverges from your actual business objective?
A research collaboration between Google Brain, Berkeley, and Stanford identified concrete safety problems in modern machine learning systems. These aren't theoretical edge cases. They're the operational gaps that turn a validated model into a compliance incident.
What This Guide Covers
This guide translates five categories of AI safety research problems into actionable controls for enterprise model risk management. You'll find requirement mappings to SR 11-7, ISO/IEC 42001:2023, and the EU AI Act, plus implementation patterns your validation team can deploy this quarter.
In scope:
- Specification failures where model objectives misalign with business intent
- Robustness gaps under distribution shift
- Assurance limitations in black-box systems
- Safe exploration constraints during model learning
- Distributional impact monitoring post-deployment
Not covered: adversarial ML attacks, privacy-preserving techniques, or AI supply chain compromise.
Key Concepts and Definitions
Negative side effects: Unintended impacts on system state or environment that occur while the model optimizes its primary objective. Example: A fraud detection model that reduces false positives by flagging fewer transactions overall, inadvertently letting sophisticated fraud through.
Reward hacking: When a model finds an unintended way to maximize its objective function without achieving the underlying goal. Your customer churn model might learn to predict "won't churn" for customers who simply haven't had time to churn yet.
Scalable oversight: The ability to evaluate model behavior as system complexity grows beyond human review capacity. If you can't audit 10,000 model decisions per day, you can't validate the model under SR 11-7 ongoing monitoring requirements.
Safe exploration: Ensuring a model doesn't cause harm while learning from production data. Reinforcement learning systems pose particular risk here.
Distributional shift: When production data diverges from training data distributions. Your model's validation evidence becomes stale the moment deployment conditions change.
Requirements Breakdown
SR 11-7 Model Risk Management
Conceptual soundness review (¶12-13): Document how your model objective maps to business intent. If you're using a proxy metric, justify why optimizing that proxy achieves the actual outcome you want. Reward hacking often hides in this gap.
Ongoing monitoring (¶15-16): Establish thresholds for distributional shift detection. When input feature distributions drift beyond your validation evidence envelope, trigger a materiality assessment. Don't wait for performance degradation.
Outcomes analysis (¶14): Track negative side effects systematically. Your model might hit its accuracy target while creating downstream operational problems you didn't instrument.
ISO/IEC 42001:2023 AI Management System
Control 6.1.2 (Risk assessment): Identify contextual risk factors that could trigger specification failures. Consider how model objectives might conflict with organizational values or stakeholder interests under edge conditions.
Control 8.2 (AI system requirements): Define explicit constraints on model behavior during exploration. If your system learns from user interactions, specify prohibited actions and implement guardrails before deployment.
Control 6.3 (Continual improvement): Establish a Plan-Do-Check-Act cycle for refining model specifications as you discover misalignments between objectives and outcomes.
EU AI Act (High-Risk AI Systems)
Article 9 (Risk management system): Your risk management must address "reasonably foreseeable misuse." Specification failures and reward hacking qualify. Document how you test for objective misalignment.
Article 15 (Accuracy, robustness, cybersecurity): Demonstrate resilience under distributional shift. Your Technical Documentation (Annex IV) should include evidence of model behavior when input distributions diverge from training data.
Article 72 (Post-Market Monitoring): Track distributional impact. If your model affects protected groups differently as production conditions evolve, your post-market surveillance must detect it.
Implementation Guidance
Build Specification Tests Early
Before model development begins, write test cases that distinguish your true objective from plausible proxies. Consider a hiring model: Does "time to productivity" mean "completes onboarding checklist quickly" or "delivers business value within six months"? Build evaluation datasets that expose this distinction.
Instrument for Negative Side Effects
Expand your model monitoring beyond accuracy metrics. Track operational indicators the model doesn't directly optimize: customer service call volume after automated decisions, manual override rates, downstream process bottlenecks. If your model improves its primary metric while degrading these, you've got a specification problem.
Establish Distributional Shift Triggers
Calculate baseline feature distributions from your validation dataset. Set thresholds (typically 2-3 standard deviations or KL divergence bounds) that trigger automatic alerts when production data drifts. Don't rely on performance monitoring alone because some shifts degrade predictions silently.
Create Oversight Leverage Points
You can't review every model decision, but you can review decision patterns. Implement stratified sampling that overweights edge cases, contested decisions, and outcomes near decision boundaries. Route 100% of decisions in the top 5% uncertainty band to human review until you've validated model behavior in that regime.
Define Exploration Boundaries
If your model updates from production feedback, specify actions it cannot take during learning. For a content recommendation system, that might mean "never recommend content from domains not in the approved list" or "never increase frequency beyond twice per day for any user." Encode these as hard constraints, not optimization penalties.
Common Pitfalls
Treating validation as one-time approval: Your validation evidence expires when production conditions change. Set calendar-based revalidation triggers (annually at minimum) plus event-based triggers for significant distributional shifts.
Optimizing proxy metrics without validating alignment: Just because a metric is measurable doesn't mean optimizing it achieves your goal. Validate the proxy-to-outcome relationship empirically before trusting model improvements.
Assuming robustness from accuracy: A model can maintain high accuracy on shifted data while making systematically different errors. Test explicitly for behavior changes under distribution shift, not just performance changes.
Skipping specification documentation: "The model predicts customer churn" isn't a specification. Document what constitutes churn, what time horizon matters, what actions the business will take based on predictions, and what outcomes you're actually trying to achieve.
Monitoring model metrics instead of business outcomes: Your model might optimize perfectly while the business suffers. Track the outcomes that matter to stakeholders, not just the metrics that matter to data scientists.
Quick Reference Table
| Safety Problem | SR 11-7 Control | ISO/IEC 42001 Control | EU AI Act Article | Implementation Pattern |
|---|---|---|---|---|
| Specification failure | Conceptual soundness (¶12-13) | 6.1.2, 8.2 | Art. 9 | Objective-outcome test cases |
| Negative side effects | Outcomes analysis (¶14) | 6.3 | Art. 72 | Multi-metric monitoring |
| Distributional shift | Ongoing monitoring (¶15-16) | 6.1.2 | Art. 15 | Shift detection thresholds |
| Reward hacking | Conceptual soundness (¶12-13) | 8.2 | Art. 9 | Proxy validation testing |
| Unsafe exploration | Development process (¶10-11) | 8.2 | Art. 15 | Hard constraint encoding |
| Scalable oversight | Ongoing monitoring (¶15-16) | 6.3 | Art. 72 | Stratified decision sampling |
Your model risk framework already has the structure to address these problems. The research collaboration between Google Brain, Berkeley, and Stanford simply made the failure modes explicit. Now you can instrument for them before they become incidents.
