Category: Adversarial Security

Data Poisoning

Also known as: AI Poisoning, Poisoning Attack

Simply put

Data poisoning is a type of cyberattack in which an adversary deliberately injects corrupted, misleading, or malicious data into the training data used to build an AI or machine learning model. Because the model learns from this tampered data, its behavior or outputs can be compromised. It is generally treated as an adversarial security threat rather than an ordinary data-quality problem.

Formal definition

Data poisoning is a form of adversarial attack in which an adversary controls or corrupts part of the training dataset used to develop a machine learning or AI model, with the intent of manipulating the resulting model's behavior. As commonly defined across the cited sources, it involves the intentional injection of corrupted or misleading data into the training pipeline. Note that the evidence provided does not detail specific attack subtypes (such as targeted versus availability attacks or backdoor insertion) or defensive controls, so those aspects are out of scope for this entry. The definitions here characterize data poisoning as a training-time threat and should not be conflated with inference-time adversarial attacks, which target a deployed model's inputs rather than its training data.

Why it matters

Data poisoning matters because it targets the foundation on which a model's behavior is built: its training data. Unlike ordinary data-quality problems that arise from noise or error, poisoning is an intentional adversarial act, meaning the corrupted data is introduced by a threat actor seeking to manipulate the resulting model. As commonly defined across the cited sources, when an adversary controls even part of the training data, they can influence how the model learns and, in turn, how it behaves once deployed. This makes data poisoning a security concern rather than solely a data-hygiene concern.

For organizations governing AI systems, this distinction has practical consequences. Poisoning is a training-time threat, which means the risk it introduces can be difficult to detect after a model has been built and put into use, because the compromised behavior is embedded in what the model has learned. This challenges assumptions in some governance and validation processes that focus primarily on a deployed model's performance rather than on the integrity of its training data pipeline. It underscores why data provenance, sourcing controls, and pipeline integrity are relevant to both AI governance and model risk considerations, even though the two disciplines address the concern from different angles.

It is important not to overstate what is established here. The evidence provided characterizes data poisoning as an adversarial attack involving the intentional injection of corrupted or misleading data, but it does not detail specific attack subtypes, defensive controls, or quantified impacts. Readers should therefore treat this entry as a functional characterization of the threat rather than a comprehensive treatment of mitigation. The specific severity and likelihood of poisoning will depend heavily on an organization's data sourcing practices and threat environment, which are out of scope for this definition.

Who it's relevant to

Data Scientists and Machine Learning Engineers

Because data poisoning targets the training dataset directly, those responsible for assembling and curating training data are on the front line of exposure. Understanding poisoning as an intentional adversarial threat, rather than an ordinary data-quality issue, informs how these practitioners approach the provenance and integrity of the data used to build a model.

AI Security and Cybersecurity Teams

Data poisoning is generally treated as an adversarial security threat, placing it within the remit of teams that assess and defend against cyberattacks. For these professionals, the relevance lies in recognizing the training pipeline as an attack surface that differs from inference-time input manipulation.

Model Risk and Validation Functions

For those concerned with the identification and control of risks arising from model use, data poisoning is relevant because it affects the integrity of what the model has learned in a way that may not be apparent from performance testing alone. This suggests that validation processes concerned with training data integrity address a distinct concern from those focused on a deployed model's outputs. Note that the evidence here does not prescribe specific validation controls.

AI Governance and Oversight Roles

Those responsible for organizational policies and accountability for AI systems may find data poisoning relevant when setting expectations around data sourcing, provenance, and pipeline controls. Governance measures can help reduce or manage this risk but should not be presented as eliminating it, and the specific controls appropriate to an organization are out of scope for this entry.

Inside Data Poisoning

Training-time attack vector

Data poisoning is a class of attack in which an adversary deliberately manipulates the data used to train or fine-tune a model, as distinct from attacks that target a model at inference time. The manipulation occurs during the data collection, labeling, or ingestion phase.

Availability poisoning

A form of poisoning that aims to degrade overall model performance or accuracy across many inputs, typically by injecting corrupted or mislabeled examples that broadly undermine the learned decision boundary.

Targeted (integrity) poisoning

A more selective form in which the attacker seeks to cause specific misclassifications or behaviors on chosen inputs while leaving general performance largely intact, which can make the effect harder to detect through aggregate metrics.

Backdoor / trigger-based poisoning

A subtype in which the attacker embeds a hidden trigger during training so that the model behaves normally on ordinary inputs but produces attacker-chosen outputs when the trigger pattern is present.

Data supply chain exposure

Poisoning risk is heightened where training data is sourced from external, crowd-sourced, scraped, or third-party pipelines, since those sources may be difficult to fully vet or control.

Relationship to model risk and governance

Data poisoning is commonly treated as a data integrity and security concern that feeds into both model risk management (as a source of erroneous or manipulated model behavior) and AI governance (as an accountability and control question over data sourcing). These perspectives overlap but are not identical and should not be collapsed.

Common questions

Answers to the questions practitioners most commonly ask about Data Poisoning.

Is data poisoning the same as a model simply performing poorly on new data?

No. Data poisoning refers to the deliberate manipulation of training or fine-tuning data to induce specific undesired model behavior, whereas general performance issues on new data typically reflect model performance degradation, distribution shift, or overfitting without any adversarial intent. Professionals frequently conflate the two because both can surface as degraded outputs, but the distinguishing feature of poisoning is an intentional adversarial action on the data pipeline. Attributing a decline to poisoning without evidence of manipulation risks misdiagnosing the underlying cause.

Does data poisoning only affect the training phase?

Not necessarily. While the term is most commonly associated with the corruption of training data, similar manipulation can, in many discussions, extend to fine-tuning datasets, data used for retrieval or augmentation, and continual or online learning pipelines that ingest new data over time. Treating poisoning as strictly a one-time training-phase concern can leave later ingestion points unmonitored. The precise scope depends on the system's architecture and data lifecycle.

How can an organization detect potential data poisoning in its pipelines?

Detection approaches commonly discussed include data provenance tracking, integrity checks and versioning on datasets, statistical anomaly detection on incoming data, and comparison of model behavior against expected baselines. No single technique is universally reliable, and detection is generally harder for subtle or targeted manipulations than for large-scale corruption. These measures reduce and help identify risk rather than eliminate it, and their effectiveness varies with the data source and threat model.

Where does responsibility for managing data poisoning risk typically sit within an organization?

Responsibilities are often distributed across lines of defense: data and model owners in the first line implement controls on data pipelines, a risk or governance function in the second line sets standards and challenges those controls, and internal audit in the third line provides independent assurance. This allocation is not universal and depends on how an organization has structured its AI governance and model risk management functions. The distinction matters because governance defines accountability while model risk management addresses the identification and control of the risk itself.

How should data poisoning risk be documented for model validation or review?

In many model risk management practices, documentation would describe the data sources, their provenance and controls, known threats to data integrity, and any testing performed to assess robustness. Validation activities may examine whether controls are appropriate to the model's inherent risk and whether residual risk after controls is acceptable. What is required in a given setting depends on applicable internal policy and any relevant guidance, which varies by jurisdiction and sector; this entry does not specify a mandatory documentation format.

What controls can help reduce data poisoning risk in ongoing operations?

Commonly cited controls include restricting and authenticating data sources, validating and sanitizing incoming data, maintaining dataset versioning and audit trails, monitoring model outputs for unexpected shifts, and limiting exposure of pipelines that ingest untrusted or external data. These are risk-reduction measures and do not guarantee prevention. The appropriate combination depends on the system's data lifecycle, threat model, and the level of inherent risk the organization is willing to accept.

Common misconceptions

Data poisoning is the same as ordinary poor data quality or accidental mislabeling.

Data quality issues are typically unintentional, whereas data poisoning as commonly defined involves deliberate, adversarial manipulation. The two can produce overlapping symptoms, but treating them identically can lead teams to overlook the adversarial threat model. Note that the boundary can blur in practice.

Standard validation performance metrics will reliably catch poisoning.

Targeted and backdoor poisoning can be designed to preserve aggregate accuracy on typical validation data while altering behavior on specific inputs or triggers, so headline performance metrics alone may not surface the problem.

Data poisoning is only a concern for models trained from scratch.

Fine-tuning, continual learning, and pipelines that ingest external or crowd-sourced data can also be exposed, since poisoning targets the training or updating process rather than a single initial build.

Best practices

Establish provenance and vetting controls over training data sources, giving particular scrutiny to external, scraped, crowd-sourced, or third-party pipelines where sourcing is harder to control.

Supplement aggregate performance validation with checks aimed at targeted and trigger-based manipulation, recognizing that headline accuracy metrics may not reveal selective poisoning.

Treat the data supply chain as an in-scope element of both model risk management and AI governance, assigning clear accountability for data integrity without conflating the two functions.

Maintain records of data ingestion, labeling, and update steps so that anomalous or suspicious contributions can be investigated and, where possible, traced.

Apply appropriate lines-of-defense separation so that those sourcing and preparing data are subject to independent review of data integrity controls.

Document the limitations of your detection and mitigation measures, framing them as steps that reduce and manage poisoning risk rather than eliminate it.