Category: Adversarial Security

Threat Modeling

Also known as: Threat Modelling

Simply put

Threat modeling is a structured way of thinking through how a technology system might be attacked, so that potential weaknesses can be spotted and addressed before they cause harm. It involves identifying the things worth protecting, the ways they could be compromised, and the safeguards that might reduce those risks. It is typically an ongoing analysis rather than a one-time check.

Formal definition

Threat modeling refers to a set of techniques used to model and analyze technology systems and services in order to identify, prioritize, and evaluate potential threats and vulnerabilities—such as structural weaknesses or missing safeguards—along with corrective measures. As commonly framed, it is a form of risk assessment that models both the attack and defense aspects of a logical entity (for example, a piece of data, an application, a host, or a system). Definitions vary across sources and communities; it is most often applied in software and cybersecurity contexts, and the specific methodology and scope differ by framework. Note that threat modeling supports risk identification and prioritization but does not by itself eliminate the modeled risks.

Why it matters

As organizations deploy AI systems, the surface area for potential attacks expands beyond traditional software concerns to include the data pipelines, models, and interfaces that support those systems. Threat modeling matters because it provides a structured way to reason about how a system might be attacked before deployment, allowing weaknesses to be identified and prioritized rather than discovered after harm has occurred. In governance terms, it supports risk identification and prioritization, which can feed into broader control selection and oversight processes.

Because threat modeling is commonly framed as a form of risk assessment that considers both the attack and defense sides of a system, it helps teams move from an ad hoc security posture toward a documented, repeatable analysis. This is valuable for audit and accountability purposes, as it produces a record of what was considered worth protecting, how it could be compromised, and what safeguards were contemplated. It is typically an ongoing analysis rather than a one-time check, which aligns with the expectation in many frameworks that risk assessment be periodically revisited as systems and their environments change.

Who it's relevant to

Security and risk assessment teams

Those responsible for evaluating how systems could be attacked use threat modeling to identify, prioritize, and evaluate potential threats and vulnerabilities, and to suggest corrective measures. It gives them a structured basis for reasoning about both the attack and defense sides of a system rather than relying on informal judgment.

Model risk managers and AI governance functions

For those overseeing AI systems, threat modeling can inform risk identification by surfacing structural weaknesses or missing safeguards in the systems and data that support models. It should be understood as a technique that supports risk assessment and prioritization, not as a control that eliminates the modeled risks. Its scope and methodology will depend on the framework adopted.

Auditors and compliance specialists

Because threat modeling is typically an ongoing analysis that documents what is worth protecting, how it could be compromised, and what safeguards are contemplated, it produces evidence that can support review and accountability. Auditors should note that definitions and methodologies vary across sources and communities, so the rigor and scope of any given threat modeling exercise depend on the framework used.

Software and system developers

As threat modeling is most often applied in software and cybersecurity contexts, developers and architects use it to spot potential weaknesses in applications, hosts, or systems before deployment, so that safeguards can be built in rather than retrofitted after an incident.

Inside Threat Modeling

System and Asset Characterization

The structured identification of what is being protected, including the model, its training and inference data, model artifacts, pipelines, and the components and boundaries of the AI system under analysis. This scoping step defines the surface over which threats are subsequently reasoned about.

Threat Identification and Enumeration

The process of identifying potential adversaries, attack vectors, and failure modes relevant to the system. For AI systems this may include data poisoning, model evasion, model extraction, membership inference, and prompt injection, among others, though the applicable set depends heavily on the model type and deployment context.

Trust Boundaries and Data Flows

The mapping of where data and control cross between components or parties of differing trust levels, often visualized through data flow representations. Trust boundaries help locate where threats are most likely to be realized and where controls are typically needed.

Risk Assessment and Prioritization

The evaluation of identified threats by factors such as likelihood and potential impact, used to prioritize which threats warrant mitigation. This assessment supports, but is distinct from, broader model risk measurement and organizational risk management activities.

Mitigations and Controls

The identification of measures intended to reduce or manage the likelihood or impact of identified threats. Threat modeling documents proposed or existing controls; as with governance controls generally, these reduce and manage risk rather than eliminate it.

Documentation and Iteration

The recording of assumptions, identified threats, decisions, and residual concerns, along with the ongoing revision of the model as the system, data, or threat environment changes. Threat modeling is commonly treated as an iterative rather than one-time exercise.

Common questions

Answers to the questions practitioners most commonly ask about Threat Modeling.

Is threat modeling the same as model risk management for AI systems?

No. Threat modeling is a security-oriented practice focused on identifying potential adversarial threats, attack surfaces, and vulnerabilities in a system, whereas model risk management—historically framed by guidance such as SR 11-7 / OCC 2011-12 in the U.S. banking context—concerns the identification, measurement, monitoring, and control of risks arising from model use, including errors, misuse, and performance issues. The two overlap where adversarial threats to a model constitute a source of model risk, but they are not interchangeable disciplines and are typically owned by different functions.

Does completing a threat modeling exercise eliminate security risk in an AI system?

No. Threat modeling is a measure that helps identify and reduce or manage risk, not one that eliminates it. It surfaces potential threats and informs mitigations, but residual risk typically remains, and its usefulness depends on the completeness of the analysis, the assumptions made, and how the system and threat landscape evolve over time. It should be treated as an ongoing rather than one-time activity.

When in the development lifecycle should threat modeling be performed?

Threat modeling is commonly performed early in the design phase so that identified threats can inform architectural and control decisions before implementation, and then revisited when significant changes occur—such as new data sources, model retraining, changed deployment context, or newly recognized attack techniques. Treating it as a recurring activity rather than a single milestone is a frequently cited practice, though specific timing depends on organizational process.

Who should be involved in threat modeling an AI system?

Threat modeling typically benefits from a cross-functional group that may include security specialists, data scientists or model developers, system architects, and, where relevant, staff familiar with governance and risk requirements. Involving people with knowledge of both the system's technical design and its intended use context helps ensure that threats are identified accurately. Exact roles vary by organization and are not standardized across frameworks.

How should the outputs of threat modeling be documented and tracked?

Outputs are commonly recorded as identified threats, associated assumptions, proposed or implemented mitigations, and any accepted residual risk, in a form that can be reviewed and updated. Linking these outputs to broader risk and governance records can support traceability and oversight. The specific documentation format and retention expectations depend on organizational policy and any applicable regulatory or standards context, which vary by jurisdiction and sector.

How does threat modeling relate to other assurance activities like validation or testing?

Threat modeling is typically an analytical exercise that identifies where threats may exist and what controls are warranted, while activities such as testing or validation assess whether the system and its controls behave as intended. Threat modeling can inform what to test—for example, by highlighting attack surfaces to probe—but it does not replace those activities. The distinction between identifying potential threats and verifying that controls work should be maintained rather than blurred.

Common misconceptions

Threat modeling is the same as a security audit or penetration test.

Threat modeling is typically a structured analytical exercise conducted to anticipate potential threats and inform design and controls, often earlier and more conceptually than a hands-on audit or penetration test. Testing activities may validate whether specific threats can be realized, but they serve a different function and do not substitute for the analytical modeling step.

Completing a threat model eliminates the modeled risks.

A threat model identifies threats and informs mitigations that reduce or manage risk; it does not remove risk. Residual risk commonly remains after controls are applied, and the model itself can be incomplete or outdated as the system and threat landscape evolve.

Threat modeling and model risk management are interchangeable for AI systems.

Threat modeling, as commonly practiced, focuses on adversarial and security-oriented threats to a system. Model risk management, as historically framed by guidance such as SR 11-7 / OCC 2011-12 in the U.S. banking context, addresses risks arising from model use more broadly, including risks from flawed development, incorrect application, and performance issues. The two can overlap for AI systems but are distinct disciplines that should not be collapsed.

Best practices

Define and document the scope, system boundaries, assets, and trust boundaries before enumerating threats, so the analysis reflects the actual deployment rather than an abstract system.

Tailor the set of considered threats to the specific model type and use context rather than applying a generic checklist, since relevant AI-specific threats vary substantially across applications.

Prioritize identified threats by an explicit assessment of likelihood and potential impact so that mitigation effort is directed where it matters most.

Record assumptions, identified threats, chosen mitigations, and any residual concerns explicitly, and treat residual risk as an outcome to be communicated rather than something eliminated.

Revisit and update the threat model iteratively as the system, its data, or the threat environment changes, rather than treating it as a one-time deliverable.

Coordinate threat modeling with, but keep it distinct from, related activities such as validation, verification, security testing, and broader model risk management, so that overlapping concerns are covered without conflating the disciplines.