Skip to main content
Bureau of AI
Category: Adversarial Security

Adversarial Machine Learning

Also known as:
Simply put

Adversarial machine learning refers to the study of attacks that attempt to deceive or manipulate machine learning systems, and of the defenses used to protect against those attacks. Attackers may feed deliberately crafted or misleading inputs to a model to cause it to behave incorrectly, or probe a system to learn about its inner workings. The term covers both the malicious techniques themselves and the defensive research aimed at identifying and reducing a model's weak points.

Formal definition

Adversarial machine learning, as commonly defined across the evidence, is the study of attacks against machine learning systems and the corresponding defenses. Reported scope includes crafting deceptive inputs intended to induce misclassification or erroneous model behavior, as well as techniques that extract information about a model's behavior and characteristics (for example, probing to infer internal properties). In many descriptions the term is used broadly, referring both to the threat activity pursued by adversaries and to the defensive practice of studying model vulnerabilities to strengthen robustness. Note that the sources differ in emphasis: one frames AML primarily as information extraction about an ML system, while others frame it as attacks-and-defenses more generally; the definition here should be understood as an umbrella concept rather than a single settled formulation. This entry describes the concept only and does not enumerate specific attack taxonomies (such as evasion, poisoning, or model extraction) beyond what the evidence supports.

Why it matters

Adversarial machine learning matters because it identifies a category of risk that traditional software security and standard model validation do not fully address. A model can perform well on typical inputs and still be manipulated by deliberately crafted or misleading inputs that cause it to behave incorrectly. This creates a gap between measured model performance under normal conditions and the model's actual robustness when facing an adversary who is actively trying to deceive it. For model risk management, this distinction is important: a model that appears accurate in validation may still carry residual risk from adversarial manipulation that ordinary performance testing would not surface.

The evidence describes adversarial machine learning as covering both the attack techniques pursued by threat agents and the defensive research used to identify a model's weak points and strengthen it. Because it spans threat activity and defensive practice, it sits at the intersection of AI security and model risk. Organizations relying on machine learning for consequential decisions have reason to understand where their models could be deceived or probed, since such weaknesses may affect the reliability of model outputs and the confidentiality of information about how a model behaves.

It should be noted that the sources emphasize different aspects of the term, and the field's definitions are not fully settled. Some framings center on extracting information about an ML system's behavior and characteristics, while others center on attacks and defenses more broadly. This entry treats adversarial machine learning as an umbrella concept rather than a single fixed definition, and it does not claim that any particular defensive measure eliminates the underlying risk; such measures reduce or manage it.

Who it's relevant to

Model risk managers
For those responsible for identifying, measuring, and controlling model risk, adversarial machine learning highlights a source of risk that may not be captured by standard performance testing. A model that performs well on normal inputs can still be susceptible to deliberately crafted inputs, meaning residual risk may remain even after conventional validation. Understanding this helps distinguish measured model performance from robustness under adversarial conditions.
AI security and information security teams
Because the evidence frames adversarial machine learning as covering both malicious threat activity and defensive practice, security teams are central to this area. Relevant concerns include inputs designed to deceive models and techniques that extract information about a model's behavior and characteristics, both of which have implications for system reliability and confidentiality.
Data scientists and model developers
Those who build and maintain machine learning systems are positioned to apply the defensive side of the discipline, which involves studying attack techniques to identify a model's weak points and strengthen it. This work sits alongside conventional development but addresses failure modes that arise specifically from adversarial manipulation rather than ordinary error.
Governance, audit, and oversight functions
For those providing independent oversight of AI systems, adversarial machine learning is relevant to assessing whether an organization has considered adversarial threats within its risk management. Given that the term's definitions are not fully settled and sources differ in emphasis, oversight functions should be attentive to how an organization scopes and frames the concept rather than assuming a single standard interpretation.

Inside AML

Evasion attacks
Manipulations of inputs at inference time designed to cause a deployed model to produce an incorrect or attacker-preferred output, typically without altering the model itself.
Poisoning attacks
Interference with the training data or training process so that the resulting model learns attacker-influenced behavior, which can affect model integrity before deployment.
Model extraction (or model stealing)
Attempts to reconstruct or approximate a model's parameters or functionality through repeated queries, often raising intellectual-property and confidentiality concerns.
Membership inference and related privacy attacks
Techniques that attempt to determine whether specific records were part of the training data or otherwise extract information about that data, implicating data-protection considerations.
Adversarial examples
Inputs perturbed in ways that are often difficult for humans to detect but that can change a model's output, commonly discussed as a manifestation of evasion attacks.
Robustness and defensive measures
Practices intended to reduce a model's susceptibility to adversarial manipulation; these measures manage rather than eliminate the underlying risk.

Common questions

Answers to the questions practitioners most commonly ask about AML.

Is adversarial machine learning the same as ordinary model performance degradation or data drift?
No, and conflating the two is a common error. Model performance degradation and data drift typically refer to a model's predictive quality declining over time due to changing real-world data distributions, without any deliberate manipulation. Adversarial machine learning, as commonly defined, concerns inputs or training data that are intentionally crafted by an actor to induce errors or otherwise exploit a model. The distinction matters because the monitoring, controls, and threat modeling appropriate to a security adversary differ from those used to detect naturally occurring drift. In practice both can be present, but they should not be treated as interchangeable.
Does defending against adversarial attacks make a model secure or eliminate the risk?
No. Adversarial defenses reduce or manage exposure rather than eliminate it. Techniques intended to increase robustness typically shift the difficulty for an attacker or narrow certain attack surfaces, but they do not provide a guarantee of security, and defenses effective against one class of attack may not generalize to others. Describing any set of controls as making a model 'secure' overstates their effect. It is more accurate to frame adversarial robustness measures as risk-reduction controls that require ongoing monitoring and reassessment.
Where does adversarial machine learning fit within model risk management versus AI governance?
The two perspectives are related but distinct. In model risk management terms, adversarial vulnerability is one source of model risk that can be identified, measured, monitored, and controlled, and it may be considered during validation and ongoing monitoring activities. In AI governance terms, it is an organizational concern that shapes policies, accountability, and oversight for how such risks are identified and escalated. The concepts overlap where governance assigns ownership for adversarial risk and model risk practices operationalize the testing, but they should not be collapsed into a single function.
How can adversarial robustness be incorporated into model validation?
In many validation approaches, adversarial considerations can be addressed as part of the broader effort to assess a model's soundness and limitations. This may include testing model behavior under deliberately perturbed or crafted inputs and documenting observed weaknesses. It is worth distinguishing validation, which typically evaluates whether a model is conceptually sound and fit for purpose, from verification, which checks that the model was implemented as intended. Adversarial testing can inform both, but the specific scope, methods, and thresholds are context-dependent and are not standardized across all frameworks.
Who should be responsible for adversarial risk across the lines of defense?
Responsibilities are often distributed rather than held by a single group. As commonly framed, the first line (those who develop and operate the model) may build in and test defensive measures, the second line (independent risk and validation functions) may challenge and assess those measures, and the third line (internal audit) may provide independent assurance over the overall process. The precise allocation depends on an organization's structure and governance model. The key point is that adversarial risk ownership should be explicitly assigned rather than assumed, without blurring the independence between these lines.
What ongoing monitoring is appropriate for adversarial risk after deployment?
Because adversarial threats can evolve, controls established at deployment are typically not sufficient on their own. Ongoing monitoring in many programs may include watching for anomalous input patterns, periodically reassessing known attack surfaces, and reviewing whether previously effective defenses remain adequate as new attack techniques emerge. This monitoring should be distinguished from general performance monitoring for drift, since the two look for different signals. The appropriate frequency and depth are context- and risk-dependent and are not defined uniformly across frameworks.
How should adversarial risk findings be documented and escalated?
Documentation practices vary, but findings from adversarial testing are commonly recorded alongside other identified model limitations so that they inform decisions about a model's use and any compensating controls. Where a governance structure exists, material findings are typically escalated through defined channels to those accountable for accepting or mitigating the risk. It is important to describe residual adversarial risk that remains after controls are applied, distinct from the inherent risk before controls, so that decision-makers understand what exposure persists. Specific documentation and escalation requirements depend on the applicable internal policies and any relevant external framework.

Common misconceptions

Adversarial machine learning is only a concern for image classifiers.
The concept, as commonly described, spans multiple model types and data modalities; framing it narrowly as an image-only problem understates the range of systems that may be exposed to manipulation.
Defensive techniques make a model immune to adversarial attacks.
Defensive measures are generally understood to reduce or manage susceptibility, not to guarantee immunity; residual risk typically remains and should be monitored rather than assumed away.
Adversarial machine learning is purely a model performance issue handled entirely within model risk management.
It sits at the overlap of model risk management and AI governance and security: risk identification and control activities intersect with organizational oversight, accountability, and security functions, and collapsing these into a single dimension can leave gaps.

Best practices

Treat adversarial risk as a distinct category during model risk identification, documenting how evasion, poisoning, extraction, and privacy-related attacks could apply to the specific model and use case.
Coordinate ownership across model risk management, AI governance, and security functions so that responsibilities for detection, response, and oversight are assigned rather than assumed.
Characterize residual adversarial risk after applying defensive measures, and avoid representing any control as eliminating the risk.
Incorporate adversarial scenarios into validation and monitoring activities so that susceptibility is assessed both before deployment and over time as inputs and threats evolve.
Protect and control access to training data and query interfaces to reduce exposure to poisoning, extraction, and inference-style attacks.
Maintain documentation of tested attack scenarios, known limitations, and out-of-scope threats so downstream reviewers understand the boundaries of the assessment.