The conventional wisdom
A growing group of researchers is advocating for independent experts to investigate AI model failures, similar to the NTSB for algorithms. The idea is appealing: as AI models become more autonomous, unbiased investigators could scrutinize failures, publish findings, and help the industry learn. Aviation and nuclear power have similar systems. So why not AI?
This argument addresses a real frustration. When an AI system fails, vendors blame deployment teams, who in turn blame training data. Everyone points to edge cases and "known limitations." No one takes responsibility, and failures repeat across organizations.
Why we disagree
Independent investigators sound promising until you consider: what would they actually investigate?
Aviation accident investigators examine physical wreckage, flight data recorders, and maintenance logs. The causal chain is clear. A hydraulic pump failed. Ice formed on the wing. The pilot misread the altimeter. These events can be traced back to specific design decisions, maintenance lapses, or training gaps.
AI model failures are different. When a credit model denies loans to qualified applicants, or a content moderation system flags legitimate speech, or a medical diagnostic tool misses early-stage disease, you're not looking at a single failure point. You're dealing with emergent behavior from millions of weighted parameters interacting with unseen data.
Independent investigators would find your training data had representation gaps, your validation suite didn't cover this input distribution, your monitoring didn't catch the drift, and your rollback procedures took too long. In other words, they'd find exactly what your internal model risk function should have identified before deployment.
The problem isn't a lack of investigation. It's that organizations deploy models without proper validation first.
The evidence
Consider SR 11-7, the Federal Reserve's guidance on model risk management. It doesn't say "deploy first, investigate failures later." It requires:
- Ongoing monitoring to verify model performance (Section III.C)
- Outcomes analysis comparing predictions to actual results (Section III.C.1)
- Ongoing validation, including benchmarking and sensitivity analysis (Section III.D)
- Clear escalation protocols when performance degrades (Section III.E)
These are regulatory expectations for any model supporting high-stakes decisions. If you're waiting for a failure to investigate your model, you've already violated the guidance.
The EU AI Act goes further. Article 61 requires providers of high-risk AI systems to report serious incidents to market surveillance authorities within 15 days. Article 72 empowers these authorities to request documentation, conduct inspections, and demand corrective action. The investigation framework already exists, starting with your Technical Documentation (Annex IV), which you should have prepared before deployment.
Aviation provides a useful parallel. The NTSB investigates crashes, but the FAA requires pre-flight inspections, maintenance schedules, pilot certifications, and black box recorders on every commercial flight. The investigation infrastructure assumes you've done the preventive work. When it finds you didn't, the consequences are severe.
What to do instead
Stop waiting for external investigators to tell you what your model risk function should already know. Build the investigative capability internally:
Establish continuous validation. Not quarterly reviews. Continuous. Your monitoring should flag distribution drift, performance degradation, and fairness metric changes in near real-time. If you're learning about model failures from user complaints or media coverage, your monitoring failed first.
Create a model incident response protocol. When performance degrades, you need defined thresholds that trigger investigation, not a Slack thread debating whether the metrics look "concerning." Document what constitutes an incident (5% drop in precision? 10% increase in false positives?), who investigates, and what remediation looks like.
Maintain validation evidence. ISO/IEC 42001 Clause 6.1.3 requires you to determine risks and opportunities related to your AI Management System. That determination requires evidence. Keep your validation test results, performance benchmarks, and fairness audits accessible. If you can't reproduce your validation findings six months later, you don't have validation, you have a PowerPoint deck.
Build post-deployment analysis into your lifecycle. The AI RMF Playbook's MEASURE function isn't a post-mortem exercise. It's ongoing assessment of how your deployed system performs against validation baselines. Track prediction errors by demographic group, input type, and decision context. When patterns emerge, investigate before they become incidents.
When the conventional wisdom is right
Independent investigators do have a role, just not the one being proposed.
Third-party model validation is useful when you need an unbiased assessment of a high-stakes model before deployment. Banks use independent validators to review credit models. Healthcare systems bring in external auditors for diagnostic algorithms. These validators aren't investigating failures. They're preventing them by challenging assumptions, testing edge cases, and verifying that your internal validation was rigorous.
Cross-industry incident sharing also adds value. When a major model failure occurs, understanding the root cause helps other organizations avoid the same mistake. But that requires the failing organization to have done rigorous internal investigation first. You can't share lessons you haven't learned.
The aviation analogy holds here: the NTSB's public reports are valuable because they synthesize findings from extensive technical investigation. They don't replace the airline's maintenance program or the FAA's oversight. They complement it.
If your organization is deploying autonomous AI systems without the validation infrastructure to investigate your own failures, independent investigators won't save you. They'll just document how badly you missed the fundamentals. Build the investigative capability now, while the failures are still hypothetical.
