The Problem: Why This Matters Now
Your security team faces a unique challenge: the threat OWASP ranks as number one for LLM applications, prompt injection, wouldn't even make the top 10 if you only counted confirmed incidents. It holds this position because mature teams invest heavily in preventing it from becoming widespread.
This gap between perceived risk and actual incidents highlights a critical aspect of LLM security. You're defending against threats that haven't materialized at scale because defenders are doing their jobs. The moment you deprioritize these controls, you'll understand their importance.
The 2025-2026 OWASP Top 10 for LLM Applications provides a community-validated threat model. This playbook translates that model into concrete defensive controls you can implement this quarter.
What You Need Before Starting
Technical Prerequisites:
- API gateway or reverse proxy with request logging (Kong, Envoy, or cloud-native equivalent)
- Centralized logging infrastructure for LLM interaction volumes
- Secret management system (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault)
- Identity and access management with granular permission controls
- Budget for LLM API costs and quota management
Organizational Prerequisites:
- Inventory of all LLM integrations across your organization (including shadow AI)
- Data classification policy identifying sensitive information
- Clear ownership assignment for each LLM application
- Incident response runbook template for LLM-specific scenarios
Documentation You'll Create:
- Approved function catalog for LLM agents
- Input sanitization rules and blocklists
- Output validation requirements
- Rate limit policies per application and user tier
Don't wait for perfect documentation. Start with one high-risk application and build your controls there.
Step-by-Step Implementation
Phase 1: Contain Prompt Injection Exposure (Weeks 1-2)
Design for Breach, Not Prevention. Assume the model's instruction boundary will be bypassed. Your goal isn't to make prompt injection impossible; it's to make it inconsequential.
Start with constraint-based architecture:
Implement Function Allowlisting for any LLM with tool access. If your customer service bot needs to query order status and update contact information, those are the only functions it should access.
Deploy Output Filtering before LLM responses reach users or downstream systems. Create a validation layer that checks for:
- Credential patterns (API keys, passwords, tokens)
- Internal URLs or IP addresses
- PII that shouldn't be in this context
- Instructions resembling prompt injection attempts
Configure your API gateway to intercept responses. For example, using Kong with a custom plugin:
-- Simplified output validation
function check_sensitive_patterns(response_body)
local patterns = {
"sk-[A-Za-z0-9]{48}", -- OpenAI API key pattern
"10\\.%d+\\.%d+\\.%d+", -- Internal IP ranges
"AKIA[0-9A-Z]{16}" -- AWS access key pattern
}
for _, pattern in ipairs(patterns) do
if string.match(response_body, pattern) then
return false, "Sensitive pattern detected"
end
end
return true
end
- Restrict Permissions at the infrastructure level. Your LLM application's service account should operate under least privilege.
Phase 2: Prevent Sensitive Information Disclosure (Weeks 2-3)
The second-ranked threat aligns closely with incident data, indicating real-world exposure of confidential data through LLM interactions.
Implement Tiered Input Controls:
Tier 1 (Blocking): Reject inputs containing high-confidence sensitive patterns before they reach the LLM.
# Pre-LLM input validation
import re
def validate_input(user_input):
high_risk_patterns = {
'ssn': r'\b\d{3}-\d{2}-\d{4}\b',
'credit_card': r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b',
'api_key': r'sk-[A-Za-z0-9]{48}'
}
for pattern_type, pattern in high_risk_patterns.items():
if re.search(pattern, user_input):
return False, f"Input contains {pattern_type}"
return True, None
Tier 2 (Redaction): Automatically redact medium-confidence patterns and log the redaction.
Tier 3 (Monitoring): Allow the input but flag it for review if it contains ambiguous sensitive content.
Configure Context Isolation. If your LLM needs to access customer data, implement retrieval-augmented generation (RAG) with strict scoping. Each query should only retrieve documents the current user is authorized to see.
Phase 3: Control Excessive Agency (Weeks 3-4)
Excessive agency has become a significant concern as teams deploy more autonomous LLM agents. These agents need constraints to prevent unauthorized actions.
Implement the Principle of Minimal Viable Functionality:
- Create a Function Registry that explicitly lists what each agent can do:
# agent-permissions.yaml
customer_service_agent:
allowed_functions:
- query_order_status
- update_shipping_address
- generate_return_label
forbidden_functions:
- delete_order
- modify_pricing
- access_payment_info
requires_human_approval:
- issue_refund # Amount threshold in function config
Implement Confirmation Workflows for high-impact actions. Require explicit user confirmation before executing database writes or external API calls.
Add Circuit Breakers that halt agent execution if it attempts prohibited actions or exceeds expected behavior patterns.
Phase 4: Manage Unbounded Consumption (Week 4)
Unbounded consumption has become a pressing issue as teams realize the resource costs of LLM applications. Without controls, a single malicious actor can drain your API budget.
Implement Quota Management:
# Rate limiting by entity
from functools import wraps
import time
def rate_limit(max_requests, time_window):
def decorator(func):
requests = {}
@wraps(func)
def wrapper(user_id, *args, **kwargs):
now = time.time()
if user_id not in requests:
requests[user_id] = []
# Clean old requests outside time window
requests[user_id] = [
req_time for req_time in requests[user_id]
if now - req_time < time_window
]
if len(requests[user_id]) >= max_requests:
raise Exception(f"Rate limit exceeded: {max_requests} requests per {time_window}s")
requests[user_id].append(now)
return func(user_id, *args, **kwargs)
return wrapper
return decorator
@rate_limit(max_requests=100, time_window=3600)
def process_llm_request(user_id, prompt):
# Your LLM call here
pass
Deploy Sandboxing to restrict what the LLM can access. Use network policies to prevent the LLM service from reaching internal APIs it doesn't need.
Validation: How to Verify It Works
Test Prompt Injection Defenses:
Run controlled prompt injection attempts against your application. Ensure it:
- Ignores unauthorized instructions
- Executes only allowed functions
- Does not reveal sensitive data
Your controls should block these attempts or make them ineffective.
Verify Sensitive Data Controls:
Submit test inputs with fake but realistic sensitive data. Confirm that:
- High-risk patterns are blocked
- Medium-risk patterns are redacted
- No sensitive data appears in logs or outputs
- Monitoring alerts trigger appropriately
Validate Rate Limits:
Use a load testing tool to exceed your quota thresholds. Confirm that:
- Requests are rejected after limits are reached
- Different user tiers have appropriate limits
- Legitimate users aren't impacted by others hitting limits
Check Agent Constraints:
Attempt to make your LLM agent perform unauthorized actions. Verify it can't execute them even if the model "wants to."
Maintenance: Ongoing Tasks
Weekly:
- Review logs for unusual patterns
- Check quota consumption trends
Monthly:
- Update sensitive data patterns
- Review and adjust output filtering rules
- Audit function allowlists
Quarterly:
- Conduct red teaming exercises
- Review OWASP's updated guidance
- Assess new LLM threats and adjust controls
- Update your incident response runbook
After Any LLM Application Change:
- Re-validate security controls
- Update documentation
- Confirm new functions appear in your allowlist
The gap between prompt injection's ranking and its incident count exists because teams like yours implement these controls. The moment you stop maintaining them, you'll close that gap from the wrong direction.
