Category: Data Governance & Quality

Data Cleansing

Also known as: Data Cleaning, Data Cleansing

Simply put

Data cleansing is the process of finding and fixing or removing information in a data set that is inaccurate, incomplete, duplicated, inconsistent, or corrupted. Its purpose is to improve the overall quality and consistency of the data so it can be relied upon for its intended use. Depending on the issue found, cleansing may involve correcting a record, standardizing its format, or removing it entirely.

Formal definition

Data cleansing (also called data cleaning) is the process of identifying and correcting or removing corrupt, inaccurate, incomplete, incorrectly formatted, duplicate, inconsistent, or irrelevant records within a dataset, table, or database to increase data quality and consistency. Typical operations include detecting and resolving errors and inconsistencies, deduplicating records, handling missing entries, and standardizing formats. The evidence provided describes cleansing at a general data-quality level and does not specify particular techniques, tooling, or governance controls; its application within model risk management or AI governance contexts (for example, as a control over training-data quality) is out of scope for these sources and should be treated separately.

Why it matters

Data cleansing addresses a foundational dependency: the reliability of any analysis, report, or model rests on the quality of the underlying data. As the source material emphasizes, cleansing exists to find and fix or remove information that is inaccurate, incomplete, duplicated, inconsistent, or corrupted so that the data can be relied upon for its intended use. Where those defects go uncorrected, downstream conclusions inherit the errors, and consistency across records cannot be assured.

Because the evidence here describes cleansing at a general data-quality level, its relevance to AI governance and model risk management should be understood by analogy rather than asserted as an established control. In many model risk frameworks, the quality of training and input data is treated as a distinct source of model risk, separate from model performance degradation over time; poor input data can undermine a model regardless of how well the model itself is specified. Data cleansing is one activity that can help manage data-quality issues, but the sources provided do not describe governance controls, validation procedures, or techniques specific to that context, and those should be treated separately.

It is worth noting that data cleansing reduces rather than eliminates data-quality risk. Correcting or removing defective records improves consistency, but it does not by itself guarantee that a dataset is fit for a particular purpose, nor does it resolve upstream questions such as whether the data was collected appropriately or represents the intended population. Those determinations are out of scope for the general definitions supplied here.

Who it's relevant to

Data Scientists and Data Engineers

Practitioners who prepare datasets for analysis or modeling rely on cleansing to detect and resolve errors, deduplicate records, handle missing entries, and standardize formats before data is used. The general definitions here describe what cleansing addresses but not the specific techniques or tooling a given team would apply.

Model Risk Managers

Input and training-data quality is commonly treated as a distinct source of model risk. Cleansing can help manage data-quality issues, but the sources provided do not frame it as a formal model risk control, and its role within a model risk management program should be defined and documented separately rather than assumed from these general data-quality definitions.

Data Quality and Data Governance Teams

Teams accountable for the overall quality and consistency of enterprise data use cleansing as one activity for improving reliability across datasets, tables, and databases. It reduces inconsistency and duplication but does not on its own establish that data is fit for a specific intended use.

Auditors and Compliance Professionals

Those reviewing how organizations manage data quality may examine whether cleansing activities occur and how they are evidenced. The definitions supplied here do not describe governance controls, so auditors should look to the organization's own procedures for how cleansing is documented and controlled.

Inside Data Cleansing

Error Detection and Correction

The identification and remediation of inaccurate, corrupt, malformed, or otherwise erroneous values in a dataset, such as invalid entries, out-of-range values, or violations of expected formats.

Handling of Missing Values

The treatment of null, blank, or absent data points through approaches such as removal, imputation, or flagging, with the chosen method typically depending on the analytical use case and the risk of introducing bias.

Deduplication

The detection and resolution of duplicate or near-duplicate records that may distort counts, aggregations, or model training distributions.

Standardization and Normalization of Formats

The alignment of values to consistent formats, units, encodings, and conventions (for example dates, categorical labels, or numeric scales) so that data can be reliably combined and compared.

Outlier and Inconsistency Identification

The examination of values that fall outside expected patterns or that conflict with related fields, followed by a decision on whether they represent genuine data, errors, or edge cases that require investigation rather than automatic removal.

Documentation of Cleansing Decisions

A record of the transformations applied, the rationale for each, and any assumptions made, supporting reproducibility, auditability, and downstream understanding of how the data was altered.

Common questions

Answers to the questions practitioners most commonly ask about Data Cleansing.

Is data cleansing the same as data validation or verification?

No. Data cleansing typically refers to detecting and correcting or removing corrupt, inaccurate, incomplete, or improperly formatted records within a dataset. Validation and verification are distinct activities: verification generally confirms that data or a process conforms to specified requirements, while validation assesses whether the data is fit for its intended purpose. Cleansing may support both, but treating them as interchangeable blurs concepts that many practitioners keep separate.

Does cleansing the training data eliminate bias in a model?

No. Cleansing addresses issues such as errors, duplicates, missing values, and formatting inconsistencies, but it does not by itself remove bias or guarantee fairness. Bias, as commonly discussed, can arise from how data was collected, sampled, or labeled, and from societal patterns reflected in otherwise 'clean' records. Cleansing may reduce certain data-quality-driven distortions, but it should not be presented as a fairness control. Bias and fairness assessment are typically treated as separate activities.

At what stage of a data pipeline is cleansing typically performed?

Cleansing is commonly applied during data preparation, often after ingestion and before feature engineering or model training. In many implementations it is not a one-time step but is repeated as new data arrives. The specific placement depends on the pipeline architecture and the intended use of the data.

How should cleansing steps be documented for model risk or governance purposes?

In many governance and model risk contexts, practitioners are expected to record what transformations were applied, the rationale for them, and their potential effect on the data. Documenting cleansing decisions supports traceability and can assist reviewers in second- or third-line functions. The level of documentation typically scales with the risk associated with the model or use case.

How do you handle missing values during cleansing without introducing new problems?

Common approaches include removing affected records, imputing values, or flagging missingness for downstream handling. Each choice can affect the resulting data distribution and, potentially, model behavior. Because these decisions can introduce their own distortions, practitioners often document the method chosen and consider its implications rather than treating any single technique as universally appropriate.

Is cleansing something that happens once, or does it need to be repeated?

In many operational settings, cleansing is treated as an ongoing activity because new data can reintroduce errors, duplicates, or formatting inconsistencies over time. Repeated or automated cleansing may be paired with monitoring, though the appropriate cadence depends on how frequently data changes and on the intended use of the dataset.

Common misconceptions

Data cleansing is the same as data validation or model validation.

Data cleansing is the process of correcting and standardizing data itself, whereas validation is a separate activity. In particular, model validation as commonly framed in model risk management concerns assessing whether a model performs as intended; conflating data cleansing with either data validation or model validation blurs distinct steps in the data and model lifecycle.

Cleansed data is error-free and therefore removes downstream risk.

Cleansing reduces certain data quality issues but does not guarantee complete accuracy, nor does it eliminate risk. Cleansing decisions such as imputation or outlier removal can themselves introduce bias or obscure signal, so residual data quality risk typically remains and should be managed rather than assumed away.

Data cleansing is a one-time preparatory task.

Because data sources, formats, and inputs can change over time, cleansing is generally an ongoing activity. Rules established at one point may become stale, and undocumented or unmonitored cleansing can contribute to data drift or inconsistency in later use.

Best practices

Document every cleansing transformation, its rationale, and its underlying assumptions so that the process is reproducible and auditable.

Distinguish cleansing from validation activities, and preserve a copy of the raw or source data so that cleansing decisions can be reviewed or reversed.

Investigate outliers and inconsistencies before removing them, since some may represent genuine but unusual observations rather than errors.

Choose missing-value handling methods deliberately, and assess whether imputation or deletion could introduce bias into downstream analysis or model training.

Apply consistent standardization rules for formats, units, and encodings, and monitor them over time as source data changes.

Treat cleansing as an ongoing, monitored process rather than a one-time step, revisiting rules when data sources or requirements evolve.