Skip to main content
Bureau of AI
Category: Data Governance & Quality

Validation Data

Also known as: Validation Set, Validation Dataset
Simply put

Validation data is a portion of a dataset that is set aside from the data used to train a machine learning model. It is used during model development to check how well the model is doing and to help make design choices, such as adjusting the model's settings, before the model is finalized. It should not be confused with data validation, which is the separate practice of checking that data is accurate and fit for use.

Formal definition

In machine learning, validation data (or a validation set) is a subset of data held back from the training data and used during training to estimate model skill and to guide design decisions such as hyperparameter tuning. As commonly defined, it is distinct from the training set (used to fit model parameters) and from the test set (used for a final, held-out performance estimate); some sources report allocations in the range of roughly 20–30% of the dataset, though split proportions vary by practice and context. Note that 'validation data' in this sense differs from 'data validation,' the process of verifying that data is clean, accurate, within permitted ranges, and ready for use. In model risk contexts, the term 'validation' can also refer to independent model validation as an organizational control, which is a separate concept not covered by the evidence here.

Why it matters

Validation data serves a distinct role in model development: it provides a check on model performance during training and informs design choices such as hyperparameter tuning before a model is finalized. Because these decisions are made against the validation set rather than the training set, the validation data helps developers estimate how well a model is likely to generalize beyond the specific examples it was fit to. Without a properly separated validation set, model developers risk making design decisions that overfit to the training data, producing a model that appears to perform well in development but behaves differently in use.

For those working in model risk and governance, the terminology around 'validation' is a frequent source of confusion, and precision matters. 'Validation data' in the machine learning sense—a held-out subset used to guide design decisions—is not the same as 'data validation,' the separate practice of verifying that data is clean, accurate, and within permitted ranges before it is used. Neither of these is the same as independent model validation as an organizational control in a model risk framework. Blurring these terms can lead to miscommunication in documentation, audit workpapers, and control descriptions, where the reader may assume a data-quality check, a development-time performance estimate, or an independent review function when a different concept was intended.

Careful documentation of how data was split—and which subset was used for training, for tuning, and for final performance estimation—supports the transparency and reproducibility that reviewers and auditors typically look for. Misusing or omitting these distinctions does not by itself create risk, but it can obscure how design choices were made and how performance was measured, which in turn can complicate later review, monitoring, and control activities.

Who it's relevant to

Data Scientists and Model Developers
Those building models rely on validation data to estimate model skill during training and to guide design decisions such as hyperparameter tuning. Maintaining a clear separation between training, validation, and test subsets helps avoid tuning decisions that overfit to the data used for development.
Model Risk Managers and Validators
Professionals reviewing model development documentation need to distinguish 'validation data' as a development-time subset from 'data validation' as a data-quality process and from independent model validation as an organizational control. Recognizing these distinctions helps ensure documentation and control descriptions are interpreted correctly.
Auditors and Compliance Officers
Reviewers examining how a model was developed benefit from clear records of how data was split and which subset informed which decisions. Precise terminology around validation supports transparency and reduces the chance that a data-quality check, a performance estimate, or an independent review is confused with one another in workpapers and reports.
Data Management and Governance Teams
Teams responsible for data pipelines should note that verifying data is clean and fit for use—data validation—is a separate practice from holding back a validation set for model development. Keeping these concepts distinct in policies and glossaries helps prevent miscommunication across development and oversight functions.

Inside Validation Data

Held-out data subset
A portion of available data set aside from the training data and used to tune or evaluate a model during development, before final testing on a separate test set. In many machine learning workflows, validation data is distinct from both the training set and the test set.
Hyperparameter tuning role
Validation data is commonly used to guide model selection and hyperparameter tuning (for example, choosing model configurations), rather than to fit the model's core parameters, which is typically done on the training data.
Performance estimation function
It provides an estimate of how a model may generalize to data not seen during training, though this estimate can be optimistic if the validation set is repeatedly used to make modeling decisions.
Distinction from model validation
In machine learning practice, 'validation data' refers to a data partition. This is conceptually separate from 'model validation' as used in model risk management contexts (such as independent review of a model's conceptual soundness and ongoing monitoring), even though both use the word 'validation.'

Common questions

Answers to the questions practitioners most commonly ask about Validation Data.

Is validation data the same as test data?
Not typically. In many machine learning workflows, validation data and test data serve distinct purposes and professionals are careful not to conflate them. Validation data is commonly used during model development to tune hyperparameters, compare candidate models, and guide selection decisions, while test data is generally held out and used only once to estimate generalization performance on unseen data. Using the terms interchangeably can lead to overly optimistic performance estimates, because a dataset repeatedly used for tuning no longer provides an unbiased assessment of how the model performs on truly novel data.
Does the term "validation data" mean the same thing as "validation" in model risk management?
No, and this is a frequent source of confusion. "Validation data" refers to a data partition used during model development, whereas "validation" in a model risk management context (as commonly framed by guidance such as SR 11-7) refers to a broader independent process of evaluating whether a model is sound, appropriate for its intended use, and functioning as intended. Model validation in that sense may involve conceptual soundness review, ongoing monitoring, and outcomes analysis, which extend well beyond the use of any single data partition. Treating the existence of a validation dataset as equivalent to having performed model validation is a common error.
How should validation data be separated from training data in practice?
Validation data is typically drawn so that it does not overlap with training data, so that tuning decisions are not evaluated on records the model has already seen. Common approaches include a held-out split or cross-validation, where portions of data rotate between training and validation roles. Practitioners generally take care to prevent leakage—situations where information from validation records influences training—since leakage can produce misleadingly favorable results. The appropriate approach often depends on data volume, the presence of time dependence, and the intended use of the model.
How large should a validation set be?
There is no single required size, and the appropriate proportion depends on factors such as total data availability, model complexity, the stability of the metrics being estimated, and any sector-specific expectations. A validation set that is too small may give noisy estimates that make model comparison unreliable, while one that is too large may leave insufficient data for training. When data is limited, cross-validation is often used instead of a fixed split to make more efficient use of available records. Documenting the rationale for the chosen approach is generally advisable for governance and review purposes.
How should time-dependent or sequential data be handled when forming validation sets?
For data with a temporal structure, randomly shuffling records into a validation set can allow future information to influence tuning, producing optimistic estimates that may not hold in deployment. Practitioners commonly use time-based splits, where validation data corresponds to a later period than training data, to better reflect how the model will encounter data in practice. The suitable method depends on the specific data and intended use, and the choice is often something that independent reviewers examine.
Should validation data and its use be documented for governance or audit purposes?
In many governance and model risk management settings, documenting how validation data was constructed, how it was used in model selection and tuning, and what steps were taken to prevent leakage supports transparency and independent review. Such documentation can help reviewers assess whether performance estimates are credible and whether the development process was sound. Specific documentation expectations vary by organization, sector, and applicable regulatory or standards regime, so requirements should be confirmed against the relevant framework rather than assumed to be uniform.

Common misconceptions

Validation data and test data are the same thing.
In many workflows they serve different roles: validation data is typically used during development to tune and select models, while a separate test set is reserved for a final, unbiased performance estimate. Blurring the two can lead to overly optimistic performance claims.
Using validation data means the model has been 'validated' in the model risk management sense.
Splitting off a validation partition of data is a data-science practice and is not equivalent to the independent model validation function described in model risk management guidance, which typically involves review of conceptual soundness, outcomes analysis, and ongoing monitoring, often by parties independent of model development.
A good validation score guarantees good real-world generalization.
Validation performance is an estimate that can be optimistic, particularly when the validation set is reused many times to make modeling decisions, or when it does not represent the population the model will encounter in deployment. It reduces uncertainty about generalization but does not eliminate it.

Best practices

Keep the validation set separate from both the training set and a final test set, and reserve the test set for a single, final performance estimate to avoid contaminating it with development decisions.
Document how the validation split was constructed, including sampling method and any assumptions about how representative it is of the intended deployment population.
Be cautious about repeatedly reusing the same validation set for many rounds of tuning, as this can produce optimistic performance estimates; consider cross-validation or a fresh evaluation set where appropriate.
Clearly distinguish, in documentation and communication, between the data-science use of 'validation data' as a partition and the model risk management concept of independent model validation, so stakeholders do not conflate the two.
Treat validation performance as an estimate of generalization rather than a guarantee, and pair it with ongoing monitoring once a model is deployed.
State the limitations and scope of the validation data explicitly, including where it may not represent sector-specific or shifting real-world conditions.