Category: Adversarial Security

Prompt Injection

Also known as: Prompt Injection Attack

Simply put

Prompt injection is a type of cyberattack that targets AI systems built on large language models (LLMs), such as chatbots and other generative AI services. An attacker crafts deceptive text and disguises it as a legitimate prompt in order to manipulate the model's behavior, for example to make it ignore its intended instructions or reveal information it should not. It is commonly described as a novel security threat specific to conversational and generative AI systems.

Formal definition

Prompt injection is a security vulnerability affecting applications built on large language models (LLMs) in which an attacker deliberately supplies crafted input designed to manipulate the model's behavior. The malicious input is disguised as, or embedded within, otherwise legitimate prompts, causing the model to deviate from its intended instructions—for example to extract data or produce unauthorized outputs. Some sources characterize it as a form of social engineering directed at conversational AI rather than at a human operator. The scope of this entry is limited to the LLM/generative-AI security concept as described in the cited evidence; the evidence does not detail specific attack subtypes (such as direct versus indirect injection), quantified prevalence, or defensive control frameworks, so those aspects are out of scope here.

Why it matters

Prompt injection matters because it targets a class of systems that organizations are deploying rapidly—applications built on large language models—and it exploits the way those systems interpret text rather than a conventional software flaw. Because the attack manipulates the model's behavior through crafted input disguised as legitimate prompts, it can cause an AI system to deviate from its intended instructions, for example by revealing information it should not or producing unauthorized outputs. For governance and risk functions, this means that an LLM-based deployment can behave in ways its operators did not intend even when the underlying infrastructure is otherwise secure.

The vulnerability is commonly described as novel and specific to conversational and generative AI systems, which is significant for teams accustomed to traditional application security controls. Some sources characterize prompt injection as a form of social engineering directed at the AI rather than at a human operator, meaning it shares conceptual features with manipulation-based attacks even though the target is a model. This framing is useful for risk teams because it signals that existing input-validation assumptions may not translate cleanly to systems that treat natural-language input as instruction.

The evidence cited here establishes the existence and general character of the threat but does not detail specific attack subtypes, quantified prevalence, or defensive control frameworks. Organizations should therefore treat this entry as a description of the concept rather than as guidance on how frequently prompt injection occurs or how to mitigate it, and should consult security-specific and up-to-date sources when designing controls.

Who it's relevant to

AI Security and Application Security Teams

Teams responsible for securing LLM-based chatbots and generative AI services need to account for prompt injection as a vulnerability that operates through crafted natural-language input rather than through conventional code-level exploits. Because this is commonly described as a novel threat specific to generative AI, existing input-validation practices may not fully address it.

Model Risk and AI Governance Functions

Governance and risk functions overseeing generative AI deployments should recognize prompt injection as a way that a model can be manipulated into deviating from its intended instructions, including revealing information it should not or producing unauthorized outputs. This is relevant to how organizations assess and monitor the behavior of LLM-based systems, though the cited evidence does not prescribe specific control frameworks.

Developers of LLM-Based Applications

Those building applications on top of large language models are directly exposed to this vulnerability, since prompt injection targets the application layer where user or third-party input is passed to the model. Awareness of the threat is a starting point, but the cited evidence does not detail defensive measures, so developers should consult dedicated security resources for mitigation.

Inside Prompt Injection

Adversarial Input Manipulation

Prompt injection typically involves crafting input text that causes a large language model or LLM-based system to disregard or override its original instructions, producing outputs the system operator did not intend.

Direct Prompt Injection

A pattern in which an end user supplies malicious or manipulative instructions directly into the prompt interface, attempting to alter the model's behavior, extract system prompts, or bypass configured constraints.

Indirect Prompt Injection

A pattern in which malicious instructions are embedded in external content the model ingests, such as retrieved documents, web pages, or tool outputs, so that the model processes attacker-controlled text without the user knowingly submitting it.

Instruction/Data Boundary Confusion

Prompt injection commonly exploits the difficulty models have in distinguishing trusted system or developer instructions from untrusted user-supplied or externally sourced data, since both are processed as natural language within the same context.

Downstream Impact Surface

The consequences that can follow a successful injection, which in many deployments include unauthorized data disclosure, unintended tool or API invocation, generation of harmful content, or circumvention of policy controls.

Common questions

Answers to the questions practitioners most commonly ask about Prompt Injection.

Is prompt injection the same as jailbreaking?

Not exactly, though the terms are often used interchangeably. As commonly defined, prompt injection refers to crafting inputs that cause a model to follow attacker-supplied instructions instead of, or in addition to, the intended ones. Jailbreaking typically refers to the narrower goal of circumventing a model's safety or content restrictions. Jailbreaking can be achieved through prompt injection techniques, but prompt injection also covers attacks aimed at other outcomes, such as exfiltrating data or hijacking downstream actions, that are not primarily about defeating content policies. Treating the two as synonyms tends to understate the broader attack surface.

Can prompt injection be fully eliminated through input filtering or better prompts?

No governance or technical control should be described as eliminating this risk. Input filtering, instruction hardening, and improved system prompts can reduce the likelihood or impact of prompt injection, but they do not remove it, in part because natural-language instructions and data are not cleanly separable and attacks can be embedded in indirect sources the model later processes. These measures are best understood as risk-reduction controls layered together rather than a solution that closes the vulnerability.

How does indirect prompt injection differ from direct prompt injection in practice?

Direct prompt injection involves an attacker supplying malicious instructions through the input channel they control, such as a chat prompt. Indirect prompt injection occurs when malicious instructions are placed in external content the model later ingests, such as a web page, document, or retrieved record, so the payload reaches the model without the attacker interacting with it directly. Practically, indirect injection expands the threat model to any data source the system consumes, which affects how you scope trust boundaries and where you apply controls.

Where does prompt injection fit within a model risk or AI governance program?

Prompt injection is generally treated as a security and operational risk associated with a deployed model-based system, so it can appear in both AI governance and model risk management workflows without collapsing the distinction between them. Governance structures typically assign accountability, policy, and oversight for how such risks are identified and escalated, while model risk activities focus on assessing and monitoring the risk arising from the model's use in context. Documenting the attack surface, controls, and residual risk supports both functions.

What role do the lines of defense play in managing prompt injection?

In many organizational frameworks, first-line owners, such as the development or product teams building the system, implement and operate controls that reduce prompt injection exposure. Second-line functions, such as risk or compliance, may set standards and independently challenge whether those controls are adequate, while third-line internal audit provides assurance over the overall process. Keeping these roles distinct helps avoid conflating the party that builds a control with the party that independently evaluates it.

How should residual prompt injection risk be documented and monitored?

Because controls reduce rather than remove exposure, it is common practice to distinguish the inherent risk of a system's design from the residual risk remaining after controls are applied, and to record that residual risk explicitly. Ongoing monitoring is typically warranted because attack techniques and the external data sources a system ingests can change over time, meaning an assessment made at deployment may not remain accurate. The specifics of monitoring frequency and thresholds tend to be sector- and system-dependent rather than governed by a single universal standard.

Common misconceptions

Prompt injection is the same as jailbreaking.

The terms overlap but are commonly distinguished. Jailbreaking generally refers to eliciting content the model is configured to refuse, while prompt injection more broadly concerns overriding or hijacking intended instructions, including via externally sourced content the user never directly typed. Treating them as identical can lead teams to overlook indirect injection vectors.

Input filtering or a well-written system prompt eliminates the risk.

Such controls can reduce exposure but do not, as commonly understood, eliminate it. Because models process instructions and data in the same natural-language channel, no single control is generally regarded as a complete defense, and layered measures are typically recommended.

Prompt injection is purely a model-quality problem to be fixed by better training.

While model behavior is a factor, prompt injection is often better understood as a systems and application-security concern involving how untrusted content enters the context, what tools and data the model can access, and how outputs are handled downstream. Framing it solely as a model-training issue can leave architectural and access-control gaps unaddressed.

Best practices

Treat all user-supplied and externally retrieved content as untrusted data rather than as trusted instructions, and design the system so that model inputs cannot silently redefine intended behavior.

Apply least-privilege principles to any tools, APIs, or data sources the model can invoke, so that a successful injection has a limited blast radius.

Layer defenses rather than relying on a single control, combining input handling, output review, and access restrictions, since no individual measure is generally regarded as complete.

Distinguish and separately test for both direct injection (via the user interface) and indirect injection (via retrieved documents, web content, or tool outputs) during evaluation.

Establish monitoring and logging of model inputs, outputs, and tool calls to support detection, investigation, and ongoing risk management of injection attempts.

Document these controls as risk-reducing measures within the organization's AI governance and model risk processes, without representing them as eliminating the risk.