You've built a governance framework, hired an AI ethics lead, and added "alignment" to your roadmap. But if your alignment work looks like feature refinement, you're not alone. The same techniques that OpenAI and Anthropic use to "align" their models with human values are often geared toward building products people will buy.
This isn't a conspiracy. It's an incentive problem, creating blind spots in how your team approaches AI risk.
Why Teams Keep Making These Mistakes
The problem starts with borrowed vocabulary. When you adopt "alignment" as a goal, you inherit a technical framing that treats human preferences as something you can model, measure, and optimize. This framing works well for product development but less so for preventing the harms your Risk Committee actually cares about.
The core issue: techniques like reinforcement learning with human feedback (RLHF) and reinforcement learning with AI feedback are designed to make models more "helpful, harmless, and honest." These sound like safety criteria but function as user satisfaction metrics. Your team ends up optimizing for the same outcomes as a customer success team, just with more sophisticated infrastructure.
Mistake 1: Treating Preference Modeling as Risk Management
Why it happens: Your team sees that major AI labs use preference models to "align" their systems. You assume this approach addresses the risks in your AI System Impact Assessment.
The real consequence: Preference modeling tells you what users find acceptable in pairwise comparisons. It doesn't tell you whether your system will cause discriminatory lending decisions, enable surveillance abuse, or fail catastrophically in edge cases. You've built a satisfaction engine, not a safety control.
Consider a team deploying a customer service model. They use RLHF to ensure responses feel "helpful and harmless." The preference model learns to avoid obvious slurs and maintain a friendly tone. It doesn't learn to refuse requests that could enable social engineering attacks, because those scenarios rarely appear in training comparisons. Your "aligned" model is optimized for customer satisfaction scores, not threat prevention.
The fix: Separate your preference optimization from your risk controls. Use ISO/IEC 23894 to structure your risk assessment around contextual risk factors, not user preferences. Your Technical Documentation (Annex IV) should list specific prohibited outputs and failure modes, then implement rule-based guardrails that preference models can't override. Preference tuning can improve user experience; it can't replace control design.
Mistake 2: Confusing Intent Alignment with Value Alignment
Why it happens: The term "intent alignment" sounds like it addresses ethical concerns. Your team assumes that if the model does what users want, you've solved the alignment problem.
The real consequence: Intent alignment means "the AI tries to do what the human wants it to do." That's a product requirement, not an ethical safeguard. If your user's intent is to generate convincing misinformation, perfect intent alignment makes your system more dangerous, not less.
The fix: Your AI Management System needs explicit value constraints that operate independently of user intent. Define your Prohibited AI Practices in governance documentation, not in model behavior. Implement these as hard constraints in your Model Provisioning process. Your model should refuse certain tasks even when the user's intent is clear and the request is well-formed. This requires policy controls, not better preference models.
Mistake 3: Outsourcing Ethics to Pairwise Comparisons
Why it happens: Your team believes that aggregating enough human judgments will converge to a universal model of "what's better." Recent research suggests this convergence is possible, so you build your safety strategy around it.
The real consequence: Pairwise comparisons capture majority preferences, not ethical principles. They encode whatever biases your annotators bring. Worse, they assume a single universal model of "better" exists. Your Stakeholder Engagement process probably revealed that different communities have conflicting values. Pairwise comparisons flatten those conflicts into a statistical average.
The fix: Use structured ethical frameworks, not crowdsourced preferences. Your Human Rights Due Diligence process should identify specific rights at risk. Your Impact Assessment (ISO/IEC 42005) should evaluate effects on vulnerable populations, not average user satisfaction. When conflicts arise, resolve them through governance decisions documented in your risk register, not through statistical aggregation of preferences.
Mistake 4: Mistaking Helpfulness for Safety
Why it happens: The "helpful, harmless, honest" triad has become shorthand for aligned AI. Your team adopts these criteria because they're what the leading labs use.
The real consequence: "Helpfulness" is a product quality metric. A model can be extremely helpful at generating phishing emails or extremely honest about how to exploit your API. These criteria optimize for user experience, not risk mitigation. Anthropic was founded by former OpenAI employees concerned about OpenAI's turn towards profit, yet both organizations use remarkably similar alignment approaches focused on these same user-satisfaction proxies.
The fix: Replace "helpfulness" metrics with risk-based performance criteria. Your Model Cards should document failure modes, not just capabilities. Your Post-Market Monitoring should track harmful outputs that users found "helpful," not just outputs they rated poorly. Define "harmlessness" in terms of specific harms enumerated in your risk assessment, not user comfort levels.
Mistake 5: Building Safety Theater Instead of Safety Controls
Why it happens: Alignment research is intellectually impressive. Your team wants to demonstrate technical sophistication. Implementing RLHF signals that you're taking safety seriously.
The real consequence: You've built a system that feels safe to use, not a system that's safe to deploy. Your preference model has learned to avoid outputs that make users uncomfortable in test scenarios. It hasn't learned to prevent the harms your regulators care about. You're optimizing for audit optics, not audit substance.
The fix: Design your validation evidence around specific risk controls, not alignment techniques. Your SR 11-7 validation should test whether the model refuses prohibited tasks, not whether it produces pleasant outputs. Your Red Teaming exercises should attempt to elicit harmful behaviors your preference model wasn't trained to prevent. Document control failures, not user satisfaction scores.
Prevention Checklist
Before your next model release, verify:
- Your risk assessment identifies specific harms, not user dissatisfaction
- Your Prohibited AI Practices are enforced through policy controls, not preference models
- Your Technical Documentation (Annex IV) enumerates failure modes your preference model doesn't address
- Your validation evidence demonstrates that safety controls work even when users want prohibited outputs
- Your Stakeholder Engagement includes populations whose values conflict with majority preferences
- Your Model Limitations and Use Restrictions specify contexts where "helpful" outputs would be harmful
- Your governance documentation separates product quality metrics from safety criteria
- Your Post-Market Monitoring tracks harmful outputs, not just low-rated outputs
The techniques that make models easier to sell rarely make them safer to deploy. Your job isn't to build aligned products. It's to build controlled systems.
