Skip to main content
Bureau of AI
Category: Adversarial Security

Model Extraction

Also known as: Model Stealing, Model Theft
Simply put

Model extraction is a type of attack in which someone tries to copy a machine learning model without authorized access to it, typically by repeatedly sending inputs to the model's public interface (such as an API) and observing the outputs. Using those query results, the attacker can build a substitute model that imitates the original's behavior. It is commonly treated as a form of intellectual property theft and a security threat to deployed AI systems.

Formal definition

Model extraction refers to an adversarial technique in which an attacker with query access to a target (victim) model systematically submits inputs and collects the corresponding outputs in order to replicate the model's functionality, typically by training a substitute or surrogate model on the query–response pairs. As commonly described in the sources here, the attack assumes only black-box API access rather than direct access to model weights, architecture, or training data. The objective may range from approximating functional behavior to reconstructing model parameters. Note that these sources characterize model extraction primarily as a security and intellectual-property threat; they do not establish standardized definitions of attack success, fidelity thresholds, or defensive controls, and treatment of these aspects varies across contexts and is out of scope for this entry.

Why it matters

Model extraction targets a valuable and often underprotected asset: the trained model itself. Organizations frequently invest substantial resources in developing proprietary models and then expose them through public interfaces such as APIs. As the sources here describe, an adversary with only black-box query access can systematically probe that interface and use the resulting input–output pairs to train a substitute model that imitates the original's behavior. This is commonly framed as both an intellectual-property threat and a security concern for deployed AI systems, because the attacker may replicate functionality without ever obtaining the model's weights, architecture, or training data.

Beyond direct loss of proprietary value, a successful extraction can serve as a stepping stone to further attacks. A high-fidelity surrogate model can, in principle, be studied offline to probe weaknesses, which may lower the cost of crafting subsequent adversarial inputs against the original system. The sources here characterize model extraction primarily as a security and intellectual-property risk; they do not establish standardized measures of attack success, fidelity thresholds, or agreed defensive controls, so organizations should treat these aspects as context-dependent rather than settled.

For governance and risk purposes, it is worth distinguishing model extraction from broader model risk. Model extraction is an adversarial security threat to a deployed asset, whereas model risk management as historically framed concerns risks arising from a model's own use and outputs. The two can intersect—for example, where a stolen or cloned model is deployed in ways that produce erroneous decisions—but they are distinct concerns and should not be collapsed into a single control objective.

Who it's relevant to

AI security and adversarial ML practitioners
Teams responsible for securing deployed models are directly concerned with model extraction as a black-box attack against exposed query interfaces. They evaluate how the attack replicates functionality through systematic querying and consider what this implies for interface design and monitoring, while recognizing that the sources here do not prescribe standardized defensive controls.
Model owners and intellectual-property stakeholders
Because model extraction is commonly treated as a form of intellectual-property theft, owners of proprietary models and the legal and business stakeholders who manage that value have a stake in understanding how a model exposed through an API can be copied without authorized access to its internals.
AI governance and risk professionals
Governance and risk specialists should account for model extraction when defining oversight and accountability for deployed AI systems. It is useful to position it as a security threat to a model asset—distinct from, though potentially intersecting with, model risk arising from a model's own outputs—so that controls are scoped correctly and not conflated.
Product and API platform teams
Teams that expose machine learning or language models through public query interfaces are relevant because such interfaces are the assumed access point for extraction. They weigh the trade-off between accessibility and exposure, noting that the sources here do not establish a single agreed set of mitigations.

Inside Model Extraction

Query-based extraction
An approach in which an adversary submits inputs to a deployed model (often through an API) and uses the returned outputs to train a substitute or surrogate model that approximates the target model's behavior.
Surrogate (substitute) model
The approximate replica an attacker builds from observed input-output pairs. It aims to mimic the target model's decision boundaries or predictions, though the degree of fidelity varies and is typically imperfect.
Attack surface
The exposure points, commonly prediction APIs, confidence scores, probability vectors, or other rich outputs, that make extraction feasible. Richer outputs generally lower the cost of extraction.
Extraction objectives
Attacker goals that commonly include stealing model functionality (to avoid licensing or development cost), enabling downstream attacks such as evasion or membership inference, or recovering proprietary or confidential model characteristics.
Fidelity versus accuracy
Two distinct measures of extraction success: fidelity refers to how closely the surrogate reproduces the target model's specific outputs or decisions, while accuracy refers to the surrogate's own task performance. These are not equivalent and can diverge.
Governance and risk relevance
Model extraction intersects both AI governance (policies on model exposure, third-party access, and intellectual-property protection) and model risk management (as a security-related risk that can amplify other model risks). The two perspectives overlap but should not be collapsed.

Common questions

Answers to the questions practitioners most commonly ask about Model Extraction.

Is model extraction the same as data exfiltration or theft of the training dataset?
No, and conflating the two is a common error. Model extraction typically refers to an adversary reconstructing or approximating a model's functionality, parameters, or decision boundaries—often through repeated querying of an exposed interface—rather than directly stealing the underlying training data. The training data may never be accessed at all. That said, extraction can raise related confidentiality concerns, and in some cases inferences about training data can follow, but the mechanisms and the assets at risk are distinct and should not be treated as interchangeable.
Does model extraction require access to the model's internal weights or source code?
Not necessarily, and assuming so understates the risk. In many discussions of this threat, extraction is described as feasible against black-box models exposed only through prediction APIs, where the adversary observes input-output behavior rather than internal parameters. The degree of fidelity an adversary can achieve varies with the model type, the interface, and the information returned (for example, class labels versus confidence scores). Internal access can make extraction easier, but it is commonly framed as a risk even without it.
Where does responsibility for addressing model extraction risk typically sit across the lines of defense?
This varies by organization and should not be assumed to fall on a single function. In many governance structures, model owners and developers in the first line address design choices and interface controls, while security and model risk functions in the second line may set standards, review controls, and monitor exposure. Internal audit as a third line may assess whether those controls operate as intended. The specific allocation depends on how an organization has scoped its AI governance and model risk responsibilities, and extraction sits at the intersection of security and model risk rather than belonging cleanly to either.
What controls are commonly discussed for reducing model extraction risk on a query interface?
Commonly cited measures include rate limiting and query monitoring to detect anomalous access patterns, restricting the granularity of returned outputs (for example returning labels rather than detailed confidence scores), authentication and access controls on the interface, and logging for later review. These are described as measures that reduce or manage the risk rather than eliminate it, and their appropriateness depends on the deployment context, the sensitivity of the model, and operational constraints such as legitimate high-volume users.
How should model extraction be reflected in a model inventory or risk assessment?
In many risk management approaches, exposure to extraction would be considered as part of a model's inherent risk profile—particularly for externally exposed or high-value models—with any implemented controls factoring into the residual risk assessment. Documenting the interface type, what outputs are exposed, and applicable safeguards can support this assessment. Because treatment is evolving and not uniformly prescribed across frameworks, organizations typically define their own criteria rather than relying on a single settled standard.
How can an organization detect that a model extraction attempt may be occurring?
Detection is generally described as difficult and imperfect. Approaches discussed include monitoring for unusual query volumes, systematic or grid-like input patterns, and access behavior inconsistent with legitimate use. These signals are indicative rather than conclusive, since legitimate high-volume or automated usage can resemble extraction activity. Detection is best understood as one layer within a broader set of controls, not a guarantee that an extraction attempt will be identified.

Common misconceptions

A model extraction attack produces an exact copy of the target model.
Extraction typically yields an approximate surrogate rather than a bit-for-bit or parameter-identical replica. Fidelity is usually partial and depends on query budget, output richness, and model complexity.
Model extraction is only an intellectual-property concern and not a security or risk-management issue.
Beyond potential IP loss, an extracted surrogate can facilitate further attacks (such as crafting evasion inputs or probing for membership inference), so it is commonly treated as a security risk with implications for both governance and model risk management.
Restricting or removing confidence scores from API outputs eliminates the risk of extraction.
Reducing output richness can raise the cost of extraction but does not eliminate it; attackers may still learn from label-only outputs given sufficient queries. Such measures reduce and manage risk rather than remove it.

Best practices

Inventory and document the model exposure points (APIs, endpoints, output formats) that could serve as an extraction attack surface, and assign clear ownership for their oversight.
Limit the richness of model outputs where feasible, for example by returning only necessary labels rather than full probability vectors, recognizing this raises attacker cost without guaranteeing protection.
Implement rate limiting, query monitoring, and anomaly detection on prediction interfaces to identify patterns consistent with systematic extraction querying.
Treat extraction as a downstream enabler of other model risks and coordinate mitigation across security, model risk management, and governance functions rather than in isolation.
Define and record measurable criteria (such as fidelity and accuracy thresholds relevant to your context) when assessing extraction exposure, so risk discussions distinguish surrogate output-matching from surrogate task performance.
Establish access controls, authentication, and contractual or policy constraints for third parties consuming model outputs to reduce unauthorized or high-volume querying.