Skip to main content
Bureau of AI
Category: Model Lifecycle & MLOps

Model Selection

Simply put

Model selection is the process of choosing the most suitable model from a set of candidate models for a given task or problem. The choice is typically made by comparing candidates against defined performance criteria to identify the one best suited to the data and objective. It is a step within model development rather than a governance or risk-control activity in itself.

Formal definition

Model selection is the task of choosing one final model from among multiple candidate models for a predictive or analytical problem, on the basis of one or more performance criteria. As commonly defined in the machine learning literature, selection criteria may include predictive performance, robustness, and model complexity evaluated against a given dataset and task. Note that selection based on performance criteria is distinct from downstream model validation and model risk management activities; while the model chosen may later be subject to independent validation and ongoing monitoring, selection alone does not establish fitness for regulatory or operational use.

Why it matters

Model selection sits at the point where design intent becomes a concrete artifact: the model that gets chosen shapes the predictions an organization will act on, the errors it will absorb, and the assumptions it will carry forward. Because selection is driven by measurable criteria such as predictive performance, robustness, and complexity, it is often treated as a purely technical optimization step. For practitioners in AI governance and model risk management, however, the more important point is what selection does not by itself establish. Choosing the best-performing candidate against a chosen metric on a given dataset does not demonstrate that the model is fit for operational or regulatory use; that determination typically comes from downstream activities such as independent validation and ongoing monitoring.

A common pitfall is to conflate a strong selection outcome with assurance. A model that scores well against the selection criterion may still perform poorly under conditions not represented in the evaluation data, may embed problematic assumptions, or may degrade over time. Framing selection as the end of the risk conversation, rather than an early input to it, can create a false sense of confidence. Under the second and third lines of defense in many model risk frameworks, the reasonableness of the selection process and the criteria used are themselves subject to challenge and review, separate from the developer's own judgment.

Selection choices are also difficult to reverse cheaply once a model is embedded in a workflow, so documenting why a candidate was chosen, what alternatives were considered, and what criteria governed the decision supports later validation, audit, and accountability. This documentation trail matters not because selection is a control in itself, but because it provides the evidence that downstream governance and risk-management activities depend on.

Who it's relevant to

Data scientists and model developers
Those building models perform selection directly, comparing candidates against defined criteria and documenting the rationale for the final choice. Their records of alternatives considered and criteria applied form the basis on which later validation and review are conducted.
Model validators (second line of defense)
Validators typically assess whether the selection criteria were appropriate and whether the chosen model is reasonable, treating selection as an input to be challenged rather than accepted at face value. This work is distinct from the developer's selection and does not by itself confirm fitness for use.
Model risk managers
Those responsible for model risk care about selection because the choice of model influences the risks an organization takes on. They rely on selection documentation to support risk identification, measurement, and monitoring, while recognizing that selection alone does not establish operational or regulatory suitability.
Internal auditors (third line of defense)
Auditors may examine whether a defensible and documented process governed model selection, and whether that process fed into downstream validation and monitoring, providing independent assurance over the surrounding controls rather than re-performing the technical selection.

Inside Model Selection

Candidate Model Space
The set of models, algorithms, or model configurations under consideration for a given task. Selection typically involves comparing across this space using defined criteria rather than choosing a single model in isolation.
Selection Criteria
The predefined metrics and considerations used to compare candidates, which commonly include predictive performance, but may also cover interpretability, stability, computational cost, fairness considerations, and fitness for the intended use. The relative weighting of these criteria is often context-dependent and is a matter of documented judgment.
Evaluation and Comparison Methodology
The procedures used to estimate and compare candidate performance, such as holdout testing, cross-validation, or out-of-time samples. This is distinct from validation of the finally selected model, which is a separate model risk management activity in many frameworks.
Fitness for Intended Use
An assessment of whether a candidate is appropriate for the specific business or regulatory purpose, data conditions, and population it will serve. A model that scores well on performance metrics may still be unsuitable if it does not align with the intended use.
Documentation and Rationale
A record of which alternatives were considered, the criteria applied, and the justification for the chosen model. In model-risk-oriented settings this documentation typically supports later review and challenge by independent parties.
Trade-off Considerations
Explicit acknowledgment of tensions between competing objectives, such as accuracy versus interpretability or complexity versus maintainability, which selection is intended to balance rather than eliminate.

Common questions

Answers to the questions practitioners most commonly ask about Model Selection.

Is model selection the same as model validation?
No. Model selection typically refers to the process of choosing among candidate models or specifications based on criteria such as predictive performance, complexity, and fit for the intended use. Model validation is a distinct activity, historically framed in guidance such as SR 11-7, that independently evaluates whether a chosen model is conceptually sound and performing as intended. Selection is often part of development; validation is commonly an independent challenge to the developed model. Conflating the two can obscure the separation of duties that many model risk management frameworks expect.
Does choosing the model with the best performance metric automatically make it the right model?
Not necessarily. A strong performance metric addresses model performance, but model selection in a risk context typically also weighs fit for the intended use, interpretability needs, data quality, robustness, and the risks introduced by added complexity. Model risk is distinct from model performance degradation, and a model that scores well on a chosen metric may still carry elevated risk if it is poorly suited to its purpose or difficult to monitor. Selection criteria are generally broader than a single accuracy or error statistic.
What criteria are commonly used to compare candidate models during selection?
Criteria vary by context, but many practitioners consider predictive performance on held-out or out-of-time data, complexity and parsimony, interpretability or explainability requirements, data availability and quality, stability, and suitability for the model's intended use. The relative weight of these criteria typically depends on the use case and applicable governance expectations. There is no single universally mandated set of selection criteria across all frameworks.
How should model selection decisions be documented?
In many model risk management practices, documentation includes the candidate models considered, the criteria applied, the rationale for the chosen model, and the alternatives rejected along with reasons. Documenting the selection rationale supports later review, independent validation, and audit. The specific documentation depth expected can vary by the model's risk tier and by organizational policy, so requirements should be confirmed against the applicable internal standards and any relevant guidance.
Who is typically involved in model selection within a lines-of-defense structure?
Model selection is generally performed by model developers, who commonly sit within the first line of defense. Depending on the framework, second-line functions such as model risk management or validation may review or challenge the selection rather than make it, preserving independence. Roles and the degree of separation vary across organizations, so the allocation of responsibility should be defined in governance policy rather than assumed.
How does model selection relate to ongoing monitoring after deployment?
Selection decisions typically set expectations that inform later monitoring, such as the performance benchmarks and assumptions that a monitoring program checks against over time. Because a selected model can experience performance degradation as data or conditions change, many practices revisit selection assumptions during periodic review. Monitoring does not eliminate model risk; it helps identify when a previously selected model may no longer be fit for its intended use.

Common misconceptions

Model selection means simply picking the model with the highest accuracy or best performance score.
Performance is commonly one criterion among several. Selection typically also weighs fitness for the intended use, interpretability, stability, cost, and other factors. A single top-line metric rarely captures whether a model is appropriate for its purpose.
Model selection is the same as, or replaces, model validation.
Selection is the process of choosing among candidates against defined criteria, while validation—as commonly framed in model risk management guidance—is a distinct activity that independently assesses whether the chosen model is conceptually sound and performs as intended. Selecting a model does not discharge validation obligations.
Once a model is selected, the choice is settled and does not need revisiting.
Selection reflects conditions at a point in time. Changes in data, populations, or intended use, as well as model performance degradation over time, may warrant re-evaluation of the earlier selection decision. Selection reduces but does not eliminate ongoing risk.

Best practices

Define and document selection criteria before comparing candidates, making explicit how performance, interpretability, stability, cost, and fitness for intended use will be weighed.
Compare candidates using a consistent evaluation methodology, and keep this comparison separate from the independent validation of the finally selected model.
Record the alternatives considered and the rationale for the chosen model so the decision can be reviewed and challenged by independent reviewers.
Assess each candidate against the specific intended use, data conditions, and population it will serve, rather than relying on aggregate performance metrics alone.
Make trade-offs explicit—such as accuracy versus interpretability or complexity versus maintainability—so decision-makers understand what is being balanced.
Establish triggers for revisiting the selection decision when data, populations, intended use, or observed performance change over time.