Category: Data Governance & Quality

Data Preparation

Also known as: Data Prep, Data Preprocessing, Data Wrangling

Simply put

Data preparation is the process of collecting, cleaning, transforming, and organizing raw data so that it is ready for analysis or further processing. It typically involves correcting errors, aligning inconsistent formats, and structuring information so it can be used reliably. It is commonly described as one of the most time-consuming stages of a data or analytics project.

Formal definition

Data preparation refers to the set of activities that transform raw source data into a form suitable for downstream analysis, modeling, or data mining. As commonly defined, it encompasses data collection, cleaning (eliminating errors and inconsistencies), transformation (aligning and standardizing formats), and, in some workflows, labeling and organizing data. In many analytics and data mining projects it is characterized as a substantial and often the most time-consuming phase; one source cites it as typically taking 50-70% of a project's time. Note that the boundaries of this term vary across sources and toolsets, and it overlaps with related terms such as data preprocessing and data wrangling without a single authoritative scope. This entry reflects general analytics and data-mining usage rather than any specific regulatory definition; data quality controls relevant to model risk management and AI governance are treated as separate, related topics.

Why it matters

Data preparation sits upstream of nearly every analytics, modeling, and AI development activity, which means its quality directly shapes the reliability of everything built on top of it. Raw source data commonly contains errors, inconsistent formats, and structural problems, and the work of collecting, cleaning, and transforming that data determines whether downstream analysis rests on a sound foundation. When preparation is rushed or poorly documented, defects introduced at this stage can propagate silently into models and reports, where they are far harder and more costly to detect.

The stage is also resource-intensive: one source characterizes data preparation as one of the most important and often time-consuming aspects of data mining, citing that it usually takes 50-70% of a project's time. That concentration of effort makes it a practical focal point for anyone budgeting or scheduling a data project, and it underscores why preparation is treated as a distinct phase rather than an afterthought.

It is worth noting the scope of this concept and its limits. As commonly defined in analytics and data-mining usage, data preparation covers collecting, cleaning, transforming, organizing, and in some workflows labeling data, but the exact boundaries vary across sources and toolsets, and the term overlaps with data preprocessing and data wrangling without a single authoritative definition. This entry reflects general practice rather than any specific regulatory definition; data quality controls that arise in model risk management and AI governance are related but separate topics and should not be assumed to be equivalent to the general practice described here.

Who it's relevant to

Data Scientists and Analysts

Practitioners who build models or perform analysis rely on prepared data as their starting point. Because collecting, cleaning, and transforming raw data can consume a large share of a project's time, understanding this phase is central to planning work and to ensuring that downstream results rest on reliable inputs.

Data Engineers and Pipeline Owners

Those responsible for moving and structuring data implement much of the collection, cleaning, and transformation work. The lack of a single authoritative scope for the term means they often define, within their own toolsets, exactly where preparation begins and ends and how it relates to adjacent steps like preprocessing and wrangling.

Project and Delivery Managers

Because data preparation is often characterized as the most time-consuming phase of a data or analytics project, managers estimating timelines and resources should account for it explicitly rather than treating it as a minor precursor to modeling.

Model Risk and AI Governance Practitioners

Data preparation is relevant background for these roles because input data quality affects model outcomes. However, the general analytics usage described here is distinct from formal data quality controls applied under model risk management or AI governance regimes; those controls are separate, related topics and should be assessed against the relevant guidance or standards rather than inferred from the general practice.

Inside Data Preparation

Data Collection and Sourcing

The process of assembling data from one or more sources for use in model development, including identifying provenance, ownership, and the conditions under which data was obtained. In many governance frameworks, documenting lineage at this stage supports later auditability and reproducibility.

Data Cleaning

The identification and treatment of errors, duplicates, missing values, and inconsistencies. As commonly defined, cleaning aims to improve data quality but does not by itself establish that the data is fit for a given modeling purpose.

Data Transformation and Feature Engineering

The conversion of raw data into structured inputs suitable for modeling, including normalization, encoding, and the construction of derived features. Choices made here can materially affect model behavior and are typically documented as part of model development records.

Data Labeling and Annotation

Where applicable, the assignment of target values or categories to observations for supervised learning. Labeling quality and the process used to produce labels are frequently a source of model risk when not controlled or reviewed.

Data Splitting

The partitioning of data into subsets such as training, validation, and testing. The distinction between the validation set used during development and independent testing used to assess generalization is often emphasized to avoid overstating performance.

Documentation and Lineage

Records describing where data came from, how it was altered, and by whom. Such documentation typically supports governance objectives such as accountability and reproducibility, and can be relevant to model validation activities.

Common questions

Answers to the questions practitioners most commonly ask about Data Preparation.

Is data preparation the same as data cleaning?

No. Data cleaning—correcting errors, handling missing values, resolving inconsistencies—is one component of data preparation, not the whole of it. Data preparation is the broader set of activities that typically also includes data collection, integration, transformation, feature engineering, labeling, splitting, and documentation. Treating the two as interchangeable is a common error that can lead teams to underinvest in the transformation and documentation steps that materially affect downstream model behavior.

Does thorough data preparation remove bias from a model?

No. Data preparation can reduce certain data-quality and representativeness issues, but it does not eliminate bias. Bias can arise from the underlying population, the measurement process, the labeling process, and modeling choices, some of which preparation cannot address. Presenting data preparation as a fix that removes bias overstates what the practice can achieve; it is more accurately described as a measure that can mitigate some sources of bias while leaving others in place. Separately, reducing measurable bias does not by itself establish fairness, which is a distinct and context-dependent concept.

How should data preparation steps be documented for governance and review purposes?

In many frameworks, documentation should capture data sources, the rationale for inclusion and exclusion decisions, transformations applied, handling of missing or anomalous values, labeling procedures, and how training, validation, and test sets were separated. The aim is reproducibility and reviewability, so that a second party—such as an independent validation function—can understand and, where relevant, reproduce the choices made. Documentation practices vary by organization and sector, so specific expectations depend on the applicable internal policy and any governing guidance.

When should data be split into training, validation, and test sets during preparation?

Splitting is commonly performed before steps that could leak information across sets, so that transformations and feature engineering learned from the training data are not fitted using validation or test data. The order and method of splitting depend on the data structure; for example, time-ordered or grouped data may require approaches that respect temporal or entity boundaries rather than a simple random split. The appropriate approach is context-specific and should be justified relative to how the model will be used.

How can data preparation choices be tested to confirm they do not distort results?

One common practice is to define preparation steps as part of a reproducible pipeline and to verify that the pipeline executes as specified—this is verification, confirming the steps do what was intended. Assessing whether the prepared data supports the model's intended use is a distinct question, closer to validation, and typically involves reviewing representativeness, checking for leakage, and evaluating sensitivity to preparation choices. Keeping these two questions separate helps avoid assuming that a correctly executed pipeline is also an appropriate one.

Who is typically responsible for data preparation within a lines-of-defense structure?

Responsibilities vary by organization, but in many arrangements the teams building and operating the model—often positioned in the first line of defense—perform data preparation, while independent review of those choices may sit with a second-line function. The specific allocation depends on how an organization defines its lines of defense and its governance policies, so this should not be treated as a fixed rule. What matters for oversight is that preparation decisions are documented and available for independent challenge.

Common misconceptions

Data preparation is a one-time step completed before modeling begins.

In practice, data preparation is often iterative and may recur as models are retrained, as data sources change, or as data drift is detected in monitoring. Treating it as a single upfront task can leave later-stage risks unaddressed.

Clean data means the data is fit for the intended model use.

Cleaning addresses errors and inconsistencies but does not by itself confirm representativeness, relevance, or suitability for a specific purpose. Data can be technically clean yet unsuitable, for example if it does not reflect the population the model will be applied to.

Data preparation is a purely technical activity separate from governance.

Choices made during preparation, such as sourcing, labeling, and transformation, can introduce or amplify model risk and raise accountability questions. Many governance frameworks treat data preparation decisions as subject to oversight and documentation rather than as purely a data-science concern.

Best practices

Document data lineage and transformations throughout the process so that data used in development can be traced, reviewed, and where feasible reproduced.

Maintain a clear separation between data used for development and data reserved for independent testing, to avoid overstating how well a model generalizes.

Assess whether prepared data is representative of and relevant to the intended use case, rather than relying on cleanliness alone as evidence of fitness.

Establish review or control processes for labeling and annotation quality, since label errors can propagate into model risk that is difficult to detect later.

Treat data preparation as potentially iterative, revisiting it when models are retrained or when monitoring indicates changes in the underlying data.

Coordinate preparation decisions with relevant governance and, where applicable, validation functions so that material choices affecting model risk are subject to appropriate oversight.