Category: Adversarial Security
Evasion Attack
Also known as: Evasion, Test-Time Evasion
Simply put
An evasion attack is a technique in which an attacker deliberately modifies inputs to a system—such as files, network traffic, prompts, or other data—so that the system fails to detect them correctly or produces the wrong result. In an AI context, this typically means making small changes to the data fed to a trained model at the time it is being used, causing it to misclassify or otherwise produce an incorrect output. The goal is usually to bypass a detection or classification mechanism without being flagged.
Formal definition
In adversarial machine learning, an evasion attack manipulates test-time (inference-time) inputs to mislead a trained model into producing incorrect predictions, commonly by applying small perturbations to input data so that the model outputs an incorrect result. In the broader network and information security sense, evasion refers to bypassing a security defense in order to deliver an exploit, attack, or malware to a target system without detection. Note that these two framings overlap but are not identical: one concerns model behavior at inference, the other concerns defeating detection controls generally. As described in the evidence, this term addresses attacks against deployed/trained models and detection systems; it does not, on the basis of the evidence provided, encompass training-time attacks (such as data poisoning) or model extraction, which are distinct adversarial techniques.
Why it matters
Evasion attacks target AI systems at the point where they are most relied upon: during live use, or inference. Because the manipulation happens to inputs after a model has been trained and deployed, an evasion attack can defeat a detection or classification control without altering the underlying model or leaving obvious traces in training data. For organizations that use machine learning as a security or screening mechanism—malware detection, fraud screening, content moderation, network intrusion detection—this means the control can appear to function normally while an adversary systematically slips past it.
Who it's relevant to
Model risk managers and validators
Evasion attacks are a reason to distinguish robustness from ordinary performance during validation. Validators typically assess whether a model performs well on representative data; evaluating resistance to deliberately manipulated inputs is a separate consideration, and evasion risk can remain as residual risk even where standard accuracy metrics look strong. This entry does not prescribe specific validation techniques, and appropriate testing will vary by model, use case, and threat context.
AI security and adversarial ML practitioners
For those building or defending machine learning systems, evasion is a core inference-time threat: attackers modify inputs to a deployed model to force misclassification or to bypass a detection or classification mechanism. Understanding evasion as an input-manipulation technique against trained models—distinct from training-time attacks such as data poisoning, which the evidence here does not cover—helps scope defensive priorities.
Security operations and detection engineering teams
Where models underpin controls such as malware, fraud, or intrusion detection, evasion attacks describe how an adversary may modify malicious inputs, files, network traffic, or prompts to bypass those controls without being flagged. This is relevant to teams that need to consider that a functioning detection model can still be defeated by adversarial inputs.
AI governance and compliance stakeholders
For those responsible for oversight of AI systems, evasion attacks are an example of an AI-specific risk that governance structures should ensure is identified, assigned ownership, and monitored. Governance controls can reduce and help manage this exposure but do not eliminate it, and the appropriate treatment depends on how critical the affected model is and the sector in which it operates.
Inside Evasion Attack
Adversarial Perturbation
A deliberate, often small modification to an input crafted so that a model produces an incorrect or attacker-desired output at inference time, while the input may appear unchanged or benign to a human observer.
Inference-Time Targeting
Evasion attacks typically occur during the model's operational (inference) phase rather than during training, distinguishing them from data poisoning attacks that corrupt the training process. This distinction should not be blurred.
White-Box vs. Black-Box Access
The attacker's knowledge of the model varies: white-box scenarios assume access to model architecture, parameters, or gradients, while black-box scenarios rely only on observing inputs and outputs. The feasibility and method of an evasion attack depend heavily on which access assumption holds.
Targeted vs. Untargeted Objective
A targeted evasion aims to force a specific incorrect output, whereas an untargeted evasion seeks any incorrect output. The two objectives imply different threat models and different defensive priorities.
Attack Surface Context
The domain in which evasion occurs (for example, image classification, malware detection, or text-based systems) shapes what a perturbation looks like and how it is constrained, so evasion techniques are not uniformly transferable across modalities.
Common questions
Answers to the questions practitioners most commonly ask about Evasion Attack.
Is an evasion attack the same as poisoning a model's training data?
No. An evasion attack, as commonly defined, targets a model at inference time by crafting or manipulating inputs to produce an incorrect or attacker-desired output, without altering the model's parameters or training data. Data poisoning instead corrupts the training process itself. Conflating the two is a frequent error; they occur at different stages of the model lifecycle and typically call for different controls.
Does an evasion attack mean the attacker has broken into the model or accessed its internals?
Not necessarily. Evasion attacks can occur in both white-box settings, where the attacker has knowledge of model internals, and black-box settings, where the attacker only interacts with inputs and observed outputs. A successful evasion attack does not by itself imply that the model's weights, code, or infrastructure have been compromised; it reflects manipulation of the input-to-output behavior. Treating every evasion attempt as evidence of a system breach can misdirect an incident response.
How can teams test whether a deployed model is susceptible to evasion attacks?
A common practice is adversarial testing, in which teams generate perturbed or crafted inputs to probe how the model responds and to estimate the conditions under which it can be induced to err. Such testing is typically conducted under controlled conditions and documented as part of model validation or ongoing monitoring. It is worth noting that adversarial testing reduces uncertainty about robustness rather than proving a model is immune; findings are generally scoped to the attack methods and threat assumptions actually exercised.
Where does responsibility for defending against evasion attacks sit within a lines-of-defense model?
Responsibilities are often distributed rather than owned by a single function. In many organizations the model development and operations owners (commonly aligned with the first line) implement input controls and monitoring, while independent validation or model risk functions (often the second line) assess robustness and challenge assumptions. The specific allocation depends on how an organization has defined its own governance structure, so this should be confirmed against internal policy rather than assumed.
What operational controls are commonly discussed for reducing exposure to evasion attacks?
Frequently cited measures include input validation and sanitization, monitoring for anomalous input patterns or output distributions, and adversarial testing during development and after deployment. These are typically framed as risk-reducing measures rather than guarantees; no single control should be presented as eliminating the possibility of a successful evasion attempt. The appropriate mix generally depends on the model's use case, threat model, and the consequences of an incorrect output.
How should evasion attack risk be documented for model risk and governance purposes?
Documentation commonly records the threat assumptions considered, the adversarial testing performed and its scope, observed weaknesses, and the controls in place to monitor and mitigate the risk. Because this is an inference-time and evolving concern, many teams also note the limitations of their testing and the conditions under which conclusions may no longer hold. The exact documentation expectations vary by organization and by any applicable framework or guidance, so requirements should be confirmed against the relevant internal and regulatory sources.
Common misconceptions
Evasion attacks and data poisoning are the same category of threat.
As commonly defined, evasion attacks operate at inference time on already-trained models, while poisoning attacks compromise the training data or process. Conflating them obscures where in the model lifecycle a control is needed.
An attacker must have full access to a model's internals to conduct an evasion attack.
Evasion can occur under both white-box and black-box conditions. In many settings, attackers craft effective perturbations using only query access to inputs and outputs, so limiting internal access does not by itself eliminate the risk.
Defenses against evasion attacks can fully eliminate the risk.
Defensive measures typically reduce or manage susceptibility to evasion rather than remove it entirely. Robustness improvements often shift or narrow the attack surface rather than close it completely.
Best practices
Define the applicable threat model explicitly, specifying whether white-box or black-box access is assumed and whether targeted or untargeted evasion is the primary concern, before selecting defenses.
Distinguish evasion (inference-time) risks from training-time risks such as poisoning in your model risk documentation, so controls are mapped to the correct point in the model lifecycle.
Evaluate model robustness against adversarial perturbations relevant to the specific input modality and deployment context, rather than assuming techniques transfer across domains.
Frame adversarial defenses as risk-reduction measures in governance and monitoring records, and avoid representing them as eliminating evasion risk.
Incorporate ongoing monitoring of inference-time inputs and outputs where feasible, since evasion susceptibility may change as the model, data, or attacker capabilities evolve.
Document the scope and limitations of any robustness testing performed, including the access assumptions and objectives tested, so downstream reviewers understand what was and was not assessed.