Category: Adversarial Security

Poisoning Attack

Also known as: Data Poisoning, Data Poisoning Attack, Training Data Poisoning

Simply put

A poisoning attack is a type of cyberattack in which someone deliberately introduces corrupted, misleading, or malicious data into the information used to train a machine learning or AI model. The goal is to distort what the model learns so that it behaves incorrectly or in ways the attacker intends once it is put into use. Because the interference happens during training rather than after deployment, the resulting problems can be difficult to detect.

Formal definition

A poisoning attack is an adversarial attack against machine learning systems in which an attacker intentionally injects, alters, or corrupts training data (or otherwise manipulates the training process) to compromise the learned model. As commonly defined, the effect manifests at model deployment or inference time, inducing degraded, biased, or attacker-controlled behavior. Poisoning is typically distinguished from attacks that operate purely at inference time (for example, evasion attacks against an already-trained model), and the term as used in this context refers to attacks on the AI/ML training pipeline. Note that 'poisoning' is also used in unrelated networking contexts (for example, DNS cache poisoning), which is a distinct concept and out of scope for this entry.

Why it matters

Poisoning attacks target a phase of the AI lifecycle that many governance and model risk frameworks have historically treated as trusted: the training data pipeline. Because the interference occurs during training rather than at inference time, the resulting harm can be embedded in the model itself and may not surface until deployment, when the model behaves in degraded, biased, or attacker-controlled ways. This makes poisoning particularly difficult to detect through conventional post-deployment monitoring that focuses on live inputs and outputs.

For organizations that rely on externally sourced, scraped, or user-contributed training data, poisoning expands the attack surface beyond the model and its runtime environment to include the provenance and integrity of the data used to build the model. From a model risk perspective, poisoning is a threat to model integrity that can undermine validation results if the corrupted data is present during both training and testing. It should be distinguished from ordinary model performance degradation, which arises from causes such as data drift rather than deliberate adversarial manipulation.

It is worth noting that the term 'poisoning' also appears in unrelated networking contexts, such as DNS cache poisoning, which is a distinct concept out of scope for AI/ML risk. Conflating the two can lead to misdirected controls; the mitigations relevant to training-data poisoning concern data governance, pipeline security, and validation rather than network-layer defenses.

Who it's relevant to

Model Risk Managers and Validators

Poisoning is a model integrity threat that can invalidate assumptions underlying validation. Because the interference happens during training and the impact appears at deployment, validators should consider whether corrupted or manipulated training data could have influenced both the model and the data used to evaluate it, rather than treating training data as inherently trustworthy.

AI Security and Adversarial ML Teams

Poisoning falls within adversarial machine learning as an attack on the training pipeline, distinct from inference-time attacks such as evasion. Security teams focused on AI systems need controls that address data provenance and pipeline integrity, and should be careful not to confuse training-data poisoning with unrelated networking uses of the term such as DNS cache poisoning.

Data Governance and Data Engineering Functions

Because poisoning depends on introducing corrupted or misleading data into the training set, functions responsible for sourcing, curating, and ingesting training data are central to reducing this risk. This is especially relevant where data is externally sourced, scraped, or contributed by users, which widens the opportunity for malicious injection.

AI Governance and Oversight Functions

Governance functions responsible for policies and accountability over AI systems should ensure that training-data integrity is addressed within the organization's control framework. Governance controls can reduce and help manage poisoning risk through data provenance requirements and pipeline oversight, but they do not eliminate it.

Inside Poisoning Attack

Training-time data manipulation

A poisoning attack, as commonly defined, involves an adversary deliberately corrupting the data used to train or fine-tune a model, so that the resulting model learns compromised behavior. This distinguishes it from attacks that manipulate inputs only at inference time.

Availability poisoning

A category in which the attacker aims to degrade overall model performance or reliability, typically by injecting noisy or mislabeled examples that reduce accuracy broadly across the model's behavior.

Targeted (integrity) poisoning

A category in which the attacker seeks to cause misbehavior on specific inputs or classes while leaving general performance largely intact, making detection through aggregate performance metrics more difficult.

Backdoor / trigger-based poisoning

A subtype in which the model is manipulated to respond in an attacker-chosen way when a specific trigger pattern is present in an input, while behaving normally otherwise. The distinction between a backdoor and a broader integrity attack is often noted by practitioners.

Attack surface: data pipeline and supply chain

The pathways through which poisoned data can enter, including third-party datasets, web-scraped corpora, crowdsourced labels, user-contributed feedback loops, and upstream pre-trained components. The relevant surface depends on how a given model is sourced and trained.

Relationship to model risk and governance

Poisoning is a security and data-integrity threat that can manifest as a model risk (compromised or unreliable model outputs) and is typically addressed through governance controls over data provenance and pipeline oversight. These are related but distinct concerns and should not be collapsed into one another.

Common questions

Answers to the questions practitioners most commonly ask about Poisoning Attack.

Is a poisoning attack the same as an evasion attack or an adversarial example?

No. These are commonly conflated but differ in when they occur in the model lifecycle. A poisoning attack manipulates the training (or fine-tuning) data or process so that the model itself learns corrupted behavior. Evasion attacks and adversarial examples typically target a model at inference time by crafting inputs designed to produce incorrect outputs without altering the trained model. Treating them as interchangeable can lead to controls being placed at the wrong stage of the pipeline.

Does a successful poisoning attack always cause visible drops in model accuracy?

Not necessarily. A common misconception is that poisoning degrades overall performance in ways monitoring would readily catch. Some poisoning techniques are designed to preserve aggregate performance metrics while introducing targeted or conditional misbehavior, such as backdoor triggers that activate only on specific inputs. This is why poisoning is often distinguished from ordinary model performance degradation, and why accuracy monitoring alone is generally not treated as sufficient detection.

At what points in the model lifecycle should controls against poisoning be considered?

Controls are commonly considered wherever training or retraining data enters the pipeline, including initial data collection, third-party or externally sourced datasets, data labeling, and ongoing or continuous retraining processes. Because poisoning targets the learning stage, points where data provenance is weak or where external contributors can influence training data are typically treated as areas of elevated attention.

How does data provenance relate to poisoning risk management?

Establishing and documenting where training data originates, how it was collected, and who could have influenced it is commonly used to help assess and reduce poisoning risk. Weak or undocumented provenance is often treated as increasing inherent risk, since it limits the ability to trace or validate the integrity of inputs. Provenance measures reduce rather than eliminate risk.

Which lines of defense typically have roles related to poisoning risk?

In many organizations using a three-lines model, the first line (those building and operating the models and data pipelines) typically implements data controls and monitoring; the second line (independent risk or validation functions) may assess whether those controls adequately address data integrity threats; and the third line (internal audit) may evaluate the overall effectiveness of the governance around them. The specific allocation of responsibilities varies by organization and sector.

How does poisoning risk intersect with model validation activities?

Validation activities may include reviewing data integrity, provenance, and the robustness of the model to manipulated training inputs, distinct from verification that the model was implemented as specified. However, standard validation focused on performance and conceptual soundness does not automatically detect poisoning, particularly targeted or backdoor forms. Whether validation scopes should be extended to address poisoning explicitly can depend on the framework, sector, and the risk profile of the model.

Common misconceptions

Poisoning attacks are the same as adversarial (evasion) attacks.

As commonly distinguished, poisoning attacks corrupt the training or fine-tuning data to compromise the model itself, whereas evasion-style adversarial attacks manipulate inputs at inference time against an already-trained model. Conflating the two obscures where controls need to be applied.

A model that performs well on standard evaluation metrics is proof that it has not been poisoned.

Targeted and backdoor poisoning can be designed to preserve normal aggregate performance while causing misbehavior only on specific inputs or triggers, so strong headline metrics do not, on their own, rule out poisoning.

Governance controls or defensive measures eliminate poisoning risk.

Controls such as data provenance tracking and pipeline oversight are typically framed as measures that reduce or manage the risk rather than remove it entirely; residual risk generally remains, particularly where training data is externally sourced.

Best practices

Establish and document data provenance for training and fine-tuning datasets, giving particular scrutiny to third-party, web-scraped, crowdsourced, and user-contributed sources.

Apply layered defenses across the data pipeline rather than relying on a single check, and treat detection as risk-reducing rather than fully protective.

Evaluate models beyond aggregate accuracy, including targeted probing for trigger-based or class-specific misbehavior that headline metrics may not surface.

Assess poisoning within the broader model risk framework, distinguishing the security threat (data integrity) from its downstream effect on model reliability, and route each to the appropriate control owner.

Maintain monitoring over feedback loops and upstream pre-trained components, since these can reintroduce poisoned data over time.

Document known limitations and residual risk in model risk assessments, noting where the attack surface depends on how the model is sourced and trained.