These questions come from procurement and third-party risk teams at banks and credit unions evaluating AI-powered core banking modernization. The questions are real, the stakes are high, and regulatory expectations are evolving faster than most vendor contracts.
How to Evaluate AI Vendors with an Outdated RFP Template
Start by splitting your evaluation into two streams: traditional system requirements and AI-specific risk controls.
Your legacy RFP criteria still matter (uptime guarantees, data residency, integration standards, disaster recovery). But you'll need to add a parallel track that addresses model risk management requirements under SR 11-7 if you're a U.S. regulated institution, or AI Management System controls from ISO/IEC 42001 Annex A if you're building a cross-border framework.
Specifically, ask vendors:
- Can you provide Model Cards documenting training data sources, known limitations, and performance benchmarks?
- What Validation Evidence can you share from independent third-party testing?
- How do you handle Model Recalibration when performance degrades post-deployment?
- What's your approach to Reproducibility if we need to audit a decision path?
If a vendor can't answer these questions with documentation, they're not ready for a regulated deployment. You're not just buying software anymore; you're onboarding a model that makes decisions about customer accounts, credit risk, or fraud detection.
Defining "Explainable AI" in Your Contract
"Explainable AI" means different things to different vendors, which is why you need to define it in your Statement of Work.
Under the EU AI Act transparency requirements for high-risk AI systems, "explainable" translates to specific Technical Documentation (Annex IV) obligations: the vendor must document how the system reaches outputs, what features drive decisions, and how you can trace individual predictions back to input data.
In your contract, specify:
- What level of explanation you need (feature importance scores, counterfactual examples, decision trees for appeals)
- Who owns the explanation tooling (is it built into the vendor platform or do you need a separate observability layer?)
- How quickly you can generate explanations during an audit or customer dispute
- Whether the vendor will support your team during regulatory examinations
Don't accept "our neural network is inherently explainable" as an answer. Require the vendor to demonstrate explanation generation during the proof-of-concept phase using your actual data scenarios.
Understanding Responsibilities in a Three-Party Risk Model
You're looking at a three-party risk model: your institution, the vendor who built the banking application, and the Foundation Model Provider supplying the underlying AI capability.
Your vendor is responsible for:
- Instructions for Use that define appropriate banking contexts
- Model Limitations and Use Restrictions documentation
- Integration testing and Validation Evidence
- Post-Market Monitoring of the combined system's performance
- Contractual flow-down of your regulatory obligations to the foundation model provider
The foundation model provider is responsible for:
- Training data governance and Annotation Quality
- Bias Mitigation at the base model level
- Rate Limiting and API stability
- Responsible Disclosure of vulnerabilities
- Compliance with General-Purpose AI Code of Practice if they're subject to the EU AI Act
You're responsible for:
- Vendor Due Diligence on both parties
- AI System Impact Assessment covering your specific use case
- Ongoing performance validation against your risk appetite
- Ensuring contractual terms create enforceable obligations all the way down the supply chain
If your vendor's contract doesn't explicitly address their relationship with the foundation model provider, you've got an AI Supply Chain Compromise risk. You need visibility into that upstream dependency.
Addressing Model Drift in Your Vendor Contract
Model drift isn't theoretical. It's a when-not-if scenario that your vendor contract needs to address explicitly.
Define Performance Thresholds in your SLA that trigger vendor notification requirements. For example: "Vendor will notify Client within 24 hours if fraud detection precision falls below 92% or if false positive rates exceed 8% in any rolling 7-day period."
Your vendor should provide:
- Automated Post-Market Monitoring dashboards you can access in real-time
- A documented Model Recalibration process with defined timelines
- Root Cause Analysis procedures when drift is detected
- Retraining data governance (whose data, what consent, what retention)
Under SR 11-7's ongoing monitoring requirements, you can't outsource accountability for model performance. Even if the vendor owns the code, you own the risk. Your contract should require the vendor to support your internal validation team's periodic reviews and provide access to model performance metrics, not just system uptime stats.
Evaluating the "AI is Always Learning" Pitch
Be skeptical. "Always learning" often means "continuously retraining without your validation team's involvement," which creates a moving target for your model inventory and validation schedule.
Ask specifically:
- Is the model retrained automatically or on a defined schedule?
- Do you get notification and approval rights before retraining?
- What's the Reproducibility guarantee if you need to recreate a model version from six months ago for an audit?
- How do you maintain Model Provisioning version control?
For most regulated use cases, you want a vendor that deploys discrete model versions on a controlled release schedule, not continuous learning in production. You need time to validate each version before it touches customer data.
If the vendor insists on continuous learning, require contractual controls: a validation holdout period, performance bounds that trigger automatic rollback, and your right to freeze the model version during regulatory examinations.
Assessing the Vendor's AI Governance
Ask if they've implemented an AI Management System aligned with ISO/IEC 42001. If they have, request evidence:
- Their AI policy and risk appetite statement
- How they apply the Plan-Do-Check-Act (PDCA) cycle to model updates
- Their internal Stakeholder Engagement process (do they consult with regulated customers before major changes?)
- Evidence of leadership accountability for AI risk
If they haven't pursued ISO/IEC 42001 certification, ask how they demonstrate governance maturity. Do they have:
- A defined AI Lifecycle Processes framework (ideally aligned with ISO/IEC 5338)?
- Red Teaming or Adversarial Simulation results they can share?
- A Responsible Disclosure program for AI-specific vulnerabilities?
- Customer references from other regulated financial institutions?
A vendor with mature AI governance will welcome these questions. A vendor who deflects them isn't ready for your risk environment.
Drafting AI Vendor Contract Language
There isn't a standard AI vendor contract template yet, which means you're building this alongside your legal team.
Start with your existing third-party risk assessment questionnaire and add an AI-specific appendix covering the topics above. The NIST AI RMF Playbook includes risk tiering guidance that can inform your vendor risk classification. ISO/IEC 42001 Annex A controls provide a checklist of governance requirements you can translate into contract obligations.
For U.S. banks, the Federal Reserve and OCC haven't published AI-specific vendor management guidance yet, but SR 11-7 principles apply: you need ongoing access to model performance data, documentation sufficient for independent validation, and contractual rights to audit the vendor's risk management practices.
Don't wait for regulatory clarity to tighten your vendor contracts. The institutions building AI procurement standards now will have a competitive advantage when the guidance does arrive.
