Skip to main content
Bureau of AI
Category: Adversarial Security

Attacker Knowledge (White-Box/Black-Box)

Also known as: White-Box Testing, Black-Box Testing, Grey-Box Testing, White-Box/Black-Box Attack Model
Simply put

Attacker knowledge describes how much information a person testing or attacking a system is assumed to have about how that system works. In a black-box scenario, the tester has little to no prior knowledge and probes the system from the outside, much like an external attacker; in a white-box scenario, the tester is given extensive internal information about the system's structure. A grey-box scenario falls in between, with partial knowledge. These distinctions matter because the level of assumed knowledge shapes what vulnerabilities a test can realistically uncover.

Formal definition

Attacker knowledge is a parameter in security and adversarial testing that defines the information available to the evaluating party about a target system, commonly framed along a spectrum from black-box to white-box, with grey-box as an intermediate case. As described in penetration testing practice, a black-box assessment simulates an external attacker with little to no prior knowledge of the target's internal structure, focusing on vulnerabilities exploitable from outside; a white-box assessment provides the tester extensive internal information and is used to simulate a targeted attack that exercises as many attack vectors as possible; grey-box combines partial internal knowledge with external probing. The evidence provided here documents these categories in the context of general penetration testing rather than AI-specific adversarial evaluation. Note that in adversarial machine learning the same terms are used with related but not identical meanings (for example, white-box often denotes access to model architecture, parameters, or gradients, while black-box denotes query-only access); those AI-specific definitions are not established by the sources cited here and should be verified against dedicated adversarial ML literature. This entry does not address the specific threat-model assumptions, taxonomies, or evaluation protocols used in AI red-teaming, which fall outside the scope of the provided evidence.

Why it matters

The assumed level of attacker knowledge directly shapes what a security assessment can and cannot reveal. A black-box test, which simulates an external attacker with little to no prior knowledge of the target's internal structure, tends to surface vulnerabilities that are exploitable from outside the system boundary. A white-box test, by contrast, provides the tester extensive internal information and is typically used to simulate a targeted attack that exercises as many attack vectors as possible. Because these two approaches probe different parts of the attack surface, the choice of knowledge model is not a mere procedural detail; it determines the coverage and interpretation of the results. Treating a black-box result as if it were exhaustive, or assuming a white-box result reflects a realistic external threat, can lead to a distorted picture of a system's exposure.

Who it's relevant to

Penetration testers and offensive security teams
These practitioners select and apply the black-box, white-box, or grey-box model as documented in penetration testing practice. Their choice governs how much internal information they receive and, consequently, which vulnerabilities the engagement is positioned to uncover.
Security and risk managers commissioning assessments
Those who scope and procure testing engagements need to understand that the knowledge model constrains coverage: a black-box test focuses on externally exploitable weaknesses, while a white-box test is suited to simulating a targeted attack across many vectors. Misunderstanding these types, as noted in the evidence, is a common error that can lead to misplaced confidence in test results.
AI red-teaming and adversarial ML practitioners (with caveats)
Teams evaluating machine learning systems encounter the same white-box/black-box vocabulary, but the AI-specific meanings—such as gradient or parameter access versus query-only access—are related but not identical to the penetration-testing definitions established by the sources here. Practitioners should verify threat-model assumptions against dedicated adversarial ML literature rather than assuming the general penetration-testing framing carries over directly.

Inside Attacker Knowledge (White-Box/Black-Box)

White-Box Access
A threat-modeling assumption in which the adversary is presumed to have full or substantial knowledge of the target model's internals, which may include architecture, learned parameters or weights, training data or its distribution, gradients, and preprocessing logic. This represents a stronger adversary assumption used to stress-test worst-case scenarios.
Black-Box Access
A threat-modeling assumption in which the adversary has no direct visibility into model internals and interacts only through inputs and observed outputs, such as querying an API and receiving predictions or scores. Attacks under this assumption typically rely on probing, output inference, or transferability from surrogate models.
Gray-Box (Partial Knowledge)
An intermediate assumption commonly used in practice, where the adversary holds some but not all information, for example knowing the model architecture or the type of preprocessing but not the exact weights. As commonly defined, the white-box/black-box distinction sits on a spectrum rather than a strict binary.
Output Granularity
The level of detail returned to the adversary, which shapes black-box feasibility. Access to confidence scores or full probability distributions typically enables stronger attacks than access to top-label predictions alone.
Threat Model Role
Attacker knowledge is one dimension of a broader threat model, alongside adversary goals and capabilities such as query budget. It is used to define assumptions when assessing robustness, not to describe an inherent property of the model itself.

Common questions

Answers to the questions practitioners most commonly ask about Attacker Knowledge (White-Box/Black-Box).

Does a black-box assessment mean the attacker has no information about the model at all?
No. Black-box typically refers to a threat model in which the attacker lacks direct access to internal parameters, architecture, gradients, or training data, but may still observe inputs and outputs and infer behavior through queries. It is a spectrum of restricted knowledge rather than a state of total ignorance. Treating black-box as "zero information" tends to understate what an adversary can learn from query access, output confidence scores, or publicly available details about the system.
Is a white-box test always more severe or more useful than a black-box test?
Not necessarily. White-box and black-box represent different assumptions about attacker knowledge rather than a strict hierarchy of severity. White-box assessments assume full or substantial access to internals and are often used to establish an upper bound on vulnerability, while black-box assessments approximate the constraints faced by an external adversary. Each answers a different question, and the more informative choice depends on the threat you are trying to characterize. Assuming white-box is universally "harder" or more relevant can lead teams to skip realistic black-box scenarios that better reflect their actual exposure.
How do we decide which knowledge assumption to use when planning an evaluation?
The choice generally follows from the threat model you are trying to represent. If you want to understand exposure to an external party with only query access, a black-box assumption is often the closer fit; if you want to stress-test the model against a fully informed adversary or an insider, a white-box assumption may be more appropriate. Many teams run assessments under more than one assumption to bracket the range of plausible attacker capabilities rather than relying on a single scenario.
What access do we typically need to provide for a white-box assessment?
White-box assessments generally assume the assessor has access to internals such as model architecture, weights or parameters, and often gradients or training details, depending on how the scenario is scoped. In practice, defining exactly what is shared is part of the exercise, since "white-box" can span a range from partial internal visibility to complete access. Documenting the specific access granted helps make results reproducible and helps others interpret the severity implied by the findings.
How should the assumed attacker knowledge be documented in evaluation results?
It is useful to state explicitly which knowledge assumption was used, what information the assessor was granted or denied, and any query or interaction limits that applied. Because black-box and white-box each cover a range of conditions, recording these details lets reviewers understand the scope of the test and avoid over-generalizing a finding to threat models it did not actually cover.
Can results from one knowledge assumption be generalized to the other?
Generally, they should not be assumed to transfer directly. A vulnerability found under white-box conditions may or may not be exploitable by a black-box adversary with limited access, and the absence of a finding under black-box constraints does not establish resilience against a fully informed attacker. Where you need assurance across multiple threat models, testing under each relevant assumption is typically more defensible than extrapolating from one to another.

Common misconceptions

Black-box access means the model is effectively safe from adversarial manipulation.
Limited internal visibility reduces but does not eliminate risk. Adversaries can often exploit query access, observed outputs, and transferability from surrogate models. Black-box is an assumption about knowledge, not a guarantee of protection.
White-box and black-box are the only two possibilities and are strictly binary.
In many threat models, attacker knowledge is best treated as a spectrum. Partial or gray-box assumptions, where an adversary knows some components but not others, are common in practical assessments.
Choosing a threat model is the same as measuring the model's actual security.
Attacker-knowledge assumptions define the conditions under which robustness is evaluated. They frame the analysis rather than certifying that the model resists real-world attackers, and conclusions are only as sound as the assumptions chosen.

Best practices

State the assumed attacker knowledge explicitly when documenting a robustness assessment, and justify why that assumption is appropriate for the deployment context.
Consider evaluating against multiple points on the knowledge spectrum, including gray-box scenarios, rather than defaulting to a single white-box or black-box assumption.
Use white-box assumptions to stress-test worst-case behavior, recognizing that results represent a stronger adversary than may typically be encountered.
Assess how much output granularity is exposed, such as confidence scores versus top-label only, and treat richer outputs as an amplifier of black-box attack feasibility.
Treat attacker knowledge as one dimension of a fuller threat model, and pair it with explicit assumptions about adversary goals and query budget.
Avoid presenting black-box conditions as evidence of safety; document them as a scoped assumption and note residual risk that remains outside the tested scenario.