Model risk teams face a practical dilemma: your existing validation framework worked for credit scorecards and pricing models, but now you're being asked to approve a customer service LLM or a document summarization tool. Do you apply SR 11-7 as written, or do you need something fundamentally different?
This isn't just a theoretical question. Your audit committee wants to know why LLM validation takes longer. Your business units want to know why you're asking for test datasets that don't exist. And your validators are quietly wondering whether their statistical testing toolkit even applies.
Treating LLMs Like Any Other Model
Some teams argue that model risk is model risk, regardless of architecture. They believe SR 11-7's three lines of defense still apply: developers validate, an independent team reviews, and ongoing monitoring catches drift.
The logic is straightforward. LLMs make predictions that affect business outcomes. They can fail in production. They require documentation, testing, and performance monitoring. Your existing model inventory already tracks hundreds of models; why create a parallel governance structure for one model type?
This approach has practical advantages. You don't need to retrain your validation team or rewrite your model risk policy. Your audit trail remains consistent. When regulators ask how you validate models, you point to the same framework you've used for years.
The framework questions still work. What's the model's intended use? What data did it train on? How do you measure performance? What happens when it fails? These questions apply whether you're validating a logistic regression or a transformer with billions of parameters.
You can test LLM outputs empirically. Measure accuracy on labeled test sets. Track error rates in production. Establish performance thresholds and trigger alerts when quality degrades. These are standard validation activities.
A Distinct LLM Evaluation Approach
Other practitioners argue that forcing LLMs into traditional validation templates creates dangerous blind spots. The differences aren't just technical; they're fundamental to how risk manifests.
Consider reproducibility. SR 11-7 assumes you can rerun a model and get the same output. But LLMs are stochastic. The same prompt can yield different responses. How do you document "expected output" when the model is designed to vary its answers?
Traditional models have defined input schemas. You know exactly what features go in. LLMs accept natural language, which means the input space is effectively infinite. You can't enumerate edge cases the way you would for a credit model with 47 features.
Performance measurement gets harder. A credit scorecard has a clear target variable and established metrics like AUC or KS statistic. What's the equivalent for an LLM that drafts emails? You can measure fluency, but can you measure whether it captured the right tone? Whether it avoided creating legal exposure?
The failure modes differ too. Traditional models typically fail gradually; performance degrades over time as data drifts. LLMs can fail catastrophically on a single prompt. They can generate plausible-sounding misinformation. They can leak training data. They can exhibit bias in ways that surface only in specific contexts.
Validation evidence looks different. For a traditional model, you review code, test data, and statistical diagnostics. For an LLM, you might need adversarial simulation results, red teaming reports, and toxicity benchmarks. Your validators need different expertise.
Where Practitioners Actually Land
Most model risk teams are adopting a hybrid approach. They keep the SR 11-7 structure but modify the validation activities.
You still require model documentation, but the content changes. Instead of documenting feature engineering logic, you document prompt engineering patterns. Instead of reviewing training code, you review fine-tuning procedures and alignment techniques. Instead of backtesting on historical data, you test on curated prompt sets designed to probe specific risks.
The three-stage lifecycle remains: development, validation, ongoing monitoring. But the techniques within each stage adapt. Development includes red teaming. Validation includes adversarial simulation. Monitoring tracks output quality metrics that didn't exist in your traditional model framework.
Some teams create LLM-specific validation templates while keeping them within the existing model risk policy. The policy principles stay the same, but the procedures acknowledge that you can't calculate a Gini coefficient for a text generator.
The approval threshold question gets interesting. Many teams are applying higher scrutiny to LLMs, not because the framework is different, but because the use cases often touch customers directly and the failure modes are less predictable.
Our Take
Treat LLMs as models subject to your existing governance structure, but don't pretend the validation techniques are the same.
The regulatory expectation hasn't changed: you need to understand what the model does, test whether it works, and monitor it in production. SR 11-7's core principle holds: the level of validation should be commensurate with risk. That principle works fine for LLMs.
But the execution must adapt. You can't validate an LLM the way you validate a logistic regression, just as you don't validate a neural network the way you validate a decision tree. The governance framework is consistent; the technical methods are model-appropriate.
This means updating your validation procedures without rewriting your model risk policy. Add sections on prompt testing, adversarial simulation, and output quality assessment. Train your validators on LLM-specific risks. Build new testing infrastructure. But keep these models in your existing inventory, approval workflow, and monitoring cadence.
The alternative creates more risk. If you treat LLMs as "not really models," they escape governance entirely. If you create a parallel approval process, you fragment accountability. Better to extend your existing framework than to pretend it doesn't apply.
The tradeoff is real: adapting your validation approach takes time and resources. Your validators need new skills. Your testing infrastructure needs new capabilities. But the cost of not adapting is higher: models in production without adequate validation evidence, audit findings you can't defend, and risk you can't measure.
