Addressing the Challenges
Conversations with model risk teams at banks reveal a consistent pattern: your processes, designed for 50-150 credit and fraud models, are now overwhelmed by the demand to deploy LLMs for various tasks. Your inventory system is outdated, your validation queue is growing, and leadership is pressing for AI readiness.
These are real concerns. They're the questions your peers are asking in working groups, on calls with consultants, and in budget meetings. Here's what you need to know and what actually works.
Q1: Do We Need to Inventory Every LLM Experiment?
Yes, but with a streamlined approach.
Your model inventory under SR 11-7 ensures you know what's in production and the associated risks. Focus on models in production. If a data scientist is testing GPT-4 prompts in a sandbox with synthetic data, it doesn't require full validation yet.
Implement a two-tier system. Tier one: a lightweight intake for experiments. Capture who's using what foundation model, what data they're touching, and what business problem they're exploring. This takes 10 minutes and is recorded in a simple form. Tier two: full inventory entry when the model moves toward production. At that point, gather vendor name, Foundation Model Provider, input data classifications, output use cases, and preliminary risk tiering (adapted from NIST AI RMF).
Avoid validating everything at the experiment stage. Focus on gating the process at the production threshold.
Q2: How Do We Validate a Model We Didn't Build?
Focus on what you control: use case, inputs, outputs, and monitoring.
SR 11-7 assumes access to model code, training data, and internal parameters. Foundation models break that assumption. Shift validation from "inspect the model" to "inspect the system."
Here's how:
- Use case validation: Ensure the intended use aligns with vendor instructions. If the Foundation Model Provider advises against high-stakes decisions without human review, don't auto-approve loans.
- Input validation: Check the data sent to the API. Is it anonymized? Does it include protected characteristics? Can you reproduce the prompt structure reliably?
- Output validation: Run adversarial simulations. Test for bias on protected classes. Check output consistency across identical prompts. Measure hallucination rates in your domain.
- Monitoring design: Decide on metrics to track post-deployment. Plan to detect drift without seeing the model's internals.
Your validation evidence should answer: do we understand and control the risks this model introduces?
Q3: Can We Use the Same Risk Tiering as for Credit Models?
You can use the same framework, but update the criteria.
Banks typically tier models based on financial exposure, regulatory scrutiny, and complexity. This still applies. A chatbot answering balance inquiries is lower risk than an LLM generating regulatory filings.
Add criteria for AI-specific risks:
- Explainability gap: Can you explain outputs to a regulator or customer? A high gap means a higher tier.
- Data sensitivity: Does the model process personal data subject to GDPR or customer data subject to GLBA? Higher sensitivity means a higher tier.
- Automation level: Is there a human in the loop, or does the system act autonomously? Full automation means a higher tier.
- Reputational exposure: Consider the impact of failure. "Bank chatbot gives wrong balance" vs. "Bank AI denies mortgages based on race" have different risk profiles.
Ensure your criteria reflect what makes an AI system risky today.
Q4: How Do We Scale Validation with a Small Team?
Build a validation system that scales through standardization and vendor due diligence.
First, standardize by model type. If deploying multiple instances of the same Foundation Model Provider's API, don't validate each separately. Validate the base model once, then focus on use-case-specific validations.
Second, rely on Vendor Due Diligence. Major providers should offer:
- Model Cards describing capabilities and limitations
- Third-party audit reports (if available)
- Documentation of their bias testing and red teaming
- Responsible disclosure processes for vulnerabilities
Your role is to assess the credibility of their documentation and model suitability for your use case.
Third, automate monitoring. You can't manually review every LLM output, but you can automate checks for prohibited content, bias indicators, and output quality metrics.
Q5: What If the Business Wants to Deploy Next Week?
You have two options: say no, or have a rapid-deployment pathway ready.
Without a lightweight validation process for low-risk models, you'll be the bottleneck. Business units might bypass you, leading to untracked deployments.
Create a rapid pathway for low-risk, non-customer-facing, human-in-the-loop use cases. Requirements:
- Approved Foundation Model Provider from your vendor list
- No personal or confidential data in prompts
- Human review of all outputs before action
- 30-day monitoring checkpoint
This is a controlled experiment with guardrails. After 30 days, review monitoring data and decide: promote to full production, continue the pilot, or shut it down.
For higher-risk use cases, the answer is "no, not next week." Remind leadership that "we deployed fast" isn't a defense when compliance questions arise.
Q6: How Do We Know If Our AI Governance Framework Works?
Measure it through metrics, incidents, and audit findings.
Track these metrics:
- Time from model intake to production (by risk tier)
- Percentage of models with complete validation evidence at deployment
- Number of models discovered in production that weren't in inventory
- Post-Market Monitoring coverage
- Mean time to detect model performance degradation
Monitor incidents:
- Model outputs that violated policy
- Models deployed without proper approval
- Vendor models sunset without your knowledge
- Monitoring gaps discovered during audits
Audit findings: If gaps are repeatedly found, your framework isn't working. The goal of ISO/IEC 42001 or an AI Management System is to catch these issues before auditors do.
Ask your validators and model owners if the process makes sense. If they're bogged down by paperwork rather than managing risk, you've built compliance theater, not a governance framework.
Next Steps
If you're building or scaling model risk management for AI:
- Start with NIST AI RMF Playbook for practical risk tiering guidance
- Review ISO/IEC 42001 Annex A controls for governance structure
- Join the Model Risk Managers' International Association working groups on AI validation
- Talk to your peers at other banks: everyone's figuring this out in real time, and the shared challenges are remarkably similar
The banks that scale successfully aren't the ones with the most sophisticated frameworks. They're the ones that build practical processes their teams can actually execute at volume.
