When Sai Rangachari, Temenos' Chief Product Officer, identified auditability, governability, and explainability as essential for AI adoption in banking, he wasn't just stating ideals. He was highlighting the operational requirements that distinguish deployable AI systems from compliance risks.
These myths persist because they're comforting. They let teams believe they can deploy AI without fundamentally changing how they validate, monitor, and explain algorithmic decisions. But comfort isn't compliance, and these misconceptions create gaps your examiners will find.
Myth 1: Explainability Is a Nice-to-Have for Customer Experience
Reality: Explainability is a regulatory requirement, not a UX enhancement.
When your credit decisioning model denies an application, you're legally obligated to provide specific reasons under fair lending laws. "The AI said no" doesn't meet the adverse action notice requirements in Regulation B. Your compliance team can't fulfill GDPR Article 22 rights to explanation with a shrug about model complexity.
This isn't about making customers feel better. It's about producing documentation that satisfies your Legal department's review before you send that denial letter. If your model can't generate decision-specific explanations tied to input features, you can't deploy it in any process that requires adverse action notices. The technical capability must exist before the business process begins, not after your first regulatory inquiry.
Myth 2: You Can Retrofit Auditability After Deployment
Reality: Audit trails must be architected into the system from day one.
Consider what "auditability" actually requires: complete lineage from training data through model versioning to individual predictions, with timestamps and approvals at each stage. You need to reconstruct exactly which model version made which decision on which data, potentially years after the fact.
You can't bolt this on. If you didn't log the training dataset's composition, you can't prove you met fair lending requirements when an examiner asks three years later. If you didn't version-control your feature engineering pipeline, you can't demonstrate reproducibility during model validation. If you didn't capture the human approval workflow for model deployment, you can't show adequate governance during an SR 11-7 review.
Your audit requirements should drive your MLOps architecture, not the other way around. Start with the question: "What will the examiner ask for?" Then build systems that generate those artifacts automatically.
Myth 3: Governability Means Having an AI Policy Document
Reality: Governance requires enforceable controls embedded in your deployment pipeline.
Your board-approved AI governance framework might check the ISO/IEC 42001 boxes on paper, but governance isn't documentation. It's the technical controls that prevent unauthorized model deployment, the approval gates that enforce validation requirements, and the monitoring systems that trigger alerts when models drift beyond approved thresholds.
If a data scientist can push a model to production without completing validation evidence requirements, your governance is fictional. If you can't automatically block deployment of models that exceed your risk tiering thresholds, you don't have governability. You have a policy that people might follow if they remember and if it's convenient.
Real governance looks like this: Your model registry won't accept a model without completed Model Cards. Your deployment pipeline checks for required approval signatures before provisioning. Your monitoring system automatically escalates when prediction distributions shift beyond control limits. These aren't manual processes that depend on compliance awareness. They're automated controls that make non-compliant deployment technically impossible.
Myth 4: Foundation Model Providers Handle Explainability for You
Reality: You own the explanation burden for every decision your system makes, regardless of whose model powers it.
When you embed a Foundation Model Provider's API into your loan origination workflow, you're still the regulated entity making credit decisions. The fact that you outsourced the model doesn't outsource your fair lending obligations. Your examiner will ask you to explain why applicant A was denied and applicant B was approved. "We used GPT and it's a black box" won't satisfy that inquiry.
This creates a specific due diligence requirement: before you select a foundation model, verify what explanation capabilities the provider offers. Can you get feature attributions for individual predictions? Can you audit the reasoning chain for specific decisions? If the answer is no, that model isn't suitable for regulated decisions, regardless of its benchmark performance.
Vendor Due Diligence for foundation models must include explanation methodology, not just accuracy metrics. Your contract should specify what audit artifacts the provider will supply and how quickly they'll respond to regulatory inquiries about specific predictions.
Myth 5: You Can Validate Explainability Methods Once and Move On
Reality: Explanation quality degrades as models drift, requiring ongoing validation.
Your SHAP values or attention weights might accurately reflect model behavior at deployment. Six months later, after your model has been retrained on new data or fine-tuned for performance, those explanations might bear no relationship to actual decision logic. Model recalibration changes the feature importance landscape. Distribution shift alters which features drive predictions.
If you're not validating explanation fidelity as part of ongoing monitoring, you're potentially providing customers and regulators with explanations that describe a model that no longer exists. This isn't a theoretical risk. Consider a team that validated their credit model's explanations in January, then retrained quarterly without re-validating explanation methods. By year-end, their adverse action notices cited features that contributed minimally to actual decisions, while omitting the features that actually drove denials.
Your post-market monitoring protocol should include explanation validation: do the explanations still accurately reflect model behavior? Test this by comparing explanation outputs to ground truth feature ablation studies. If explanations drift, your model needs re-validation before you generate another customer-facing decision rationale.
What to Do Instead
Start with your regulatory obligations and work backward to technical requirements. List every situation where you must explain a decision, audit a model's behavior, or demonstrate governance to an examiner. Then design systems that generate those artifacts as a byproduct of normal operation.
Build your model registry as an audit system first and a convenience tool second. Require Model Cards that document limitations, Instructions for Use that specify approved contexts, and Technical Documentation that maps to your validation evidence. Make these requirements enforceable through deployment pipeline controls.
For foundation models, create a vendor assessment framework that evaluates explanation capabilities alongside performance metrics. Require contractual commitments about audit support and explanation methodology documentation.
Most importantly, stop treating these three elements as separate workstreams. Auditability, governability, and explainability aren't parallel tracks. They're interconnected requirements that must be architected into every AI system you deploy. Your audit trail proves your governance worked. Your governance enforces your explainability requirements. Your explainability enables your audit evidence.
The banks that get AI right won't be the ones with the most sophisticated models. They'll be the ones whose compliance teams can answer examiner questions with system-generated evidence instead of apologetic explanations about technical complexity.
