Category: Validation & Testing

Cross-Validation

Also known as: CV, k-fold cross-validation, rotation estimation

Simply put

Cross-validation is a technique used to check how well a machine learning model is likely to perform on data it has not seen before, helping to guard against overfitting. It works by repeatedly splitting the available data into portions used to train the model and portions used to test it, then combining the results. Note that this statistical technique is distinct from the broader model validation process used in model risk management, which typically encompasses conceptual soundness, governance, and ongoing monitoring beyond performance estimation alone.

Formal definition

Cross-validation is a resampling-based model evaluation method commonly used to estimate a model's prediction error on unseen data and to support model selection. In a typical k-fold implementation, the dataset is partitioned into k subsets (folds); the model is trained on k-1 folds and evaluated on the held-out fold, with the process rotated so each fold serves once as the validation set, and the resulting error estimates are aggregated. As commonly framed in the literature, it is intended to provide a less optimistically biased estimate of generalization performance than evaluating on training residuals alone. Practitioners should note that its statistical behavior is complex and, per recent research, not fully understood, and that what cross-validation estimates (and how reliably) depends on assumptions about the data and the estimation target. This entry addresses cross-validation as a technical performance-estimation method; it does not by itself constitute the full model validation activity described in model risk management guidance, which is broader in scope.

Why it matters

Cross-validation matters because a model that appears highly accurate on the data it was trained on may perform poorly on new data, a failure mode commonly referred to as overfitting. By estimating prediction error on data the model has not seen during training, cross-validation gives practitioners a less optimistically biased picture of likely generalization performance than evaluating on training residuals alone. For teams responsible for model quality, this makes it a foundational tool for model selection and for deciding whether a candidate model is fit to advance toward deployment.

It is important to keep cross-validation in its proper scope. As a statistical performance-estimation technique, it addresses one dimension of model quality—how well predictions may generalize—but it does not by itself constitute the broader model validation activity described in model risk management guidance, which typically also encompasses conceptual soundness, governance, documentation, and ongoing monitoring. Treating a favorable cross-validation result as sufficient evidence that a model has been 'validated' in the model risk management sense is a common and consequential error; the two uses of the word 'validation' are distinct and should not be conflated.

Practitioners should also recognize that cross-validation's statistical behavior is complex and, according to recent research, not fully understood. What it estimates, and how reliably, depends on assumptions about the data and the target of estimation. Because of this, cross-validation results should be interpreted with appropriate caution rather than treated as a definitive measure of a model's real-world reliability.

Who it's relevant to

Data Scientists and Model Developers

Those building machine learning models use cross-validation to estimate generalization performance, guard against overfitting, and compare candidate models during model selection. They are also best positioned to recognize that the technique's reliability depends on data assumptions and the estimation target, and to interpret results accordingly.

Model Validators and Model Risk Managers

Validators encounter cross-validation as one piece of evidence within a broader model validation exercise. It can inform judgments about performance, but it does not substitute for the wider scope of model risk management activity—such as assessing conceptual soundness, governance, and ongoing monitoring. Distinguishing the statistical technique from the full validation process is essential to avoid overstating what a cross-validation result demonstrates.

Auditors and Reviewers

Those reviewing model development documentation may look at whether cross-validation was applied appropriately and whether its results were interpreted with suitable caution. Auditors should be alert to the common error of treating a favorable cross-validation outcome as equivalent to comprehensive model validation in the model risk management sense.

Inside CV

Data Partitioning

The process of splitting a dataset into complementary subsets, where some portions are used to train a model and others are held back to evaluate it. Cross-validation systematically rotates which portions serve each role.

Folds

The equally sized (or approximately equal) partitions into which data is divided. In k-fold cross-validation, the data is split into k folds, with each fold serving once as the evaluation set while the remaining folds are used for training.

Resampling Iteration

The repeated training-and-evaluation cycle across different fold assignments. Each iteration produces a performance estimate, and the results are typically aggregated (for example, averaged) to yield a more stable overall estimate.

Performance Estimation

The central purpose of the technique: producing an estimate of how a model is expected to perform on data it was not trained on, intended to reduce dependence on any single train-test split.

Variants

Common forms include k-fold, stratified k-fold (preserving class proportions across folds), leave-one-out, and time-series-aware schemes that respect temporal ordering. The appropriate variant depends on data structure and the problem context.

Common questions

Answers to the questions practitioners most commonly ask about CV.

Does cross-validation validate a model in the model risk management sense?

No, and conflating the two is a common error. Cross-validation is a resampling technique used during model development to estimate how a model's performance may generalize to unseen data. Model validation, as commonly framed in model risk management guidance such as SR 11-7, is a broader, independent process that typically includes evaluation of conceptual soundness, data quality, ongoing monitoring, and outcomes analysis. Cross-validation may inform one aspect of performance testing within a validation exercise, but it does not by itself constitute model validation.

Does a good cross-validation result mean the model will perform well in production?

Not necessarily. Cross-validation estimates generalization based on the available dataset, so its reliability depends on whether that data represents the conditions the model will face in deployment. It does not account for distributional shift, changes in population over time, or model performance degradation after deployment. A strong cross-validated result is an estimate of expected performance under the sampled conditions, not a guarantee of production outcomes, and ongoing monitoring is typically still needed.

How do I choose the number of folds for k-fold cross-validation?

The choice typically involves a trade-off between bias and variance in the performance estimate and the computational cost of refitting the model multiple times. Common practice uses values such as 5 or 10 folds, but the appropriate choice depends on dataset size, model training cost, and the stability of the estimate you need. This entry does not prescribe a single correct value, as the suitable configuration is context-dependent.

How should cross-validation handle time-series or temporally ordered data?

Standard random-split cross-validation can be inappropriate for temporally ordered data because it may allow future observations to inform predictions about the past, leaking information. Practitioners commonly use time-aware schemes that preserve chronological ordering so that training data precedes the data used for evaluation. The specific approach should reflect the temporal structure of the problem and the way the model will be used.

How do I prevent data leakage when applying preprocessing steps within cross-validation?

A common pitfall is fitting preprocessing steps, such as scaling, imputation, or feature selection, on the full dataset before splitting. This can leak information from the evaluation folds into training and produce optimistic performance estimates. To reduce this risk, preprocessing steps are typically fit only on the training portion of each fold and then applied to the corresponding held-out portion, for example by encapsulating them within a pipeline.

What should cross-validation results be used for, and what remains out of scope?

Cross-validation is commonly used to estimate generalization performance, compare candidate models, and support hyperparameter selection. It is out of scope for confirming production readiness on its own, for assessing fairness or bias beyond what the sampled data reveals, and for substituting the independent, holistic review associated with formal model validation. Its outputs are estimates that should be interpreted alongside other evidence rather than treated as definitive.

Common misconceptions

Cross-validation guarantees that a model will perform well on future, real-world data.

Cross-validation estimates generalization performance under the assumption that the evaluation data resembles data the model will encounter. It does not eliminate the risk of degradation from distribution shift, data leakage, or conditions not represented in the dataset. In practice it reduces reliance on a single split rather than providing a guarantee.

Cross-validation is a substitute for a fully independent, held-out validation of the model.

Cross-validation is a resampling technique for estimating performance and supporting model selection or tuning; in a model risk management sense it is one input to validation, not the whole of it. Independent evaluation on data untouched during model development, along with conceptual soundness review and ongoing monitoring, are typically treated as distinct activities that cross-validation alone does not replace.

More folds always produce a better or more reliable estimate.

Increasing the number of folds changes the bias-variance characteristics and computational cost of the estimate rather than uniformly improving it. The suitable number of folds and the choice of variant depend on dataset size, structure, and the intended use of the estimate.

Best practices

Choose a cross-validation variant that matches the data structure, for example using stratified folds for imbalanced classification and time-aware schemes for temporally ordered data rather than random splits.

Fit all data-dependent preprocessing steps (such as scaling, imputation, or feature selection) inside each training fold only, to avoid leakage of information from evaluation folds into training.

Report the dispersion of fold results, not only the aggregated average, so that the stability of the estimate can be assessed.

Preserve an independent held-out dataset that is untouched during cross-validation, tuning, and model selection, and treat it as separate from the cross-validation process.

Document the folding scheme, number of folds, random seeds, and any grouping constraints so the estimation procedure is reproducible and reviewable.

Frame cross-validation results as one input to model evaluation, and pair them with ongoing performance monitoring given that estimates may not hold under distribution shift.