Scope - What This Guide Covers
This guide focuses on securing instructions for large language models (LLMs) in production. You'll learn how to protect system prompts from prompt injections, jailbreaks, and instruction override attacks.
The guide covers:
- Instruction hierarchy design patterns
- Implementing privileged instructions
- Validation checkpoints for adversarial resistance
- Security controls compatible with SR 11-7 model risk frameworks
This isn't about general LLM security or data privacy. It's specifically about preventing adversaries from replacing your model's core instructions with malicious prompts.
Key Concepts and Definitions
Prompt Injection: An attack where user input contains instructions that override the model's system-level directives. The model can't reliably distinguish between your instructions and the adversary's.
Instruction Hierarchy: A security architecture that assigns privilege levels to different instruction sources. System-level prompts receive higher trust than user inputs or retrieved content.
Privileged Instructions: Core directives that define the model's purpose, constraints, and security boundaries. These should resist modification regardless of user input.
Jailbreak: A technique that manipulates the model into ignoring safety guardrails or operational constraints through carefully crafted prompts.
Requirements Breakdown
Model Risk Management Context
If you're operating under SR 11-7 or similar frameworks, instruction security falls under operational controls and ongoing monitoring. Your model validation evidence should demonstrate:
- Defined instruction boundaries: Document which instructions are privileged and why.
- Override resistance testing: Show that adversarial prompts don't bypass core constraints.
- Monitoring for instruction drift: Detect when model behavior deviates from intended instructions.
Technical Security Requirements
Your instruction hierarchy needs three layers:
Layer 1: Immutable System Context
Instructions that define the model's core purpose and hard constraints. These never change based on user input or runtime context.
Layer 2: Dynamic Operational Parameters
Context-specific instructions that adapt to the user's role, session, or task but still maintain security boundaries.
Layer 3: User Input
The lowest-privilege layer. Content here should never escalate to override Layers 1 or 2.
Implementation Guidance
Design Your Privilege Model
Start by categorizing every instruction source in your deployment:
- System prompts you control
- Retrieved documents or knowledge base content
- User messages
- Tool outputs or API responses
- Fine-tuning data (if you're using a custom model)
Assign each source a trust level. System prompts get the highest privilege. User messages get the lowest. Everything else falls between based on your threat model.
Build Instruction Separation
Don't concatenate privileged instructions with user input in a single prompt. Use your platform's native separation mechanisms:
- OpenAI's system/user message roles
- Anthropic's system context
- Custom delimiters with explicit privilege markers
If your platform doesn't support native separation, you'll need compensating controls. Consider a validation layer that checks outputs against your privileged instruction set before returning results.
Implement Validation Checkpoints
Add checkpoints that verify the model's adherence to privileged instructions:
Pre-execution validation: Before processing user input, confirm it doesn't contain instruction-like patterns that could trigger override behavior.
Post-execution validation: After the model generates output, verify it complies with your core constraints. If the output violates privileged instructions, reject it and log the incident.
Continuous monitoring: Track patterns where user inputs correlate with unexpected model behavior. This signals potential injection attempts.
Test Against Known Attack Patterns
Your validation evidence should include Adversarial Simulation results. Build a test suite covering:
- Direct instruction override attempts ("Ignore previous instructions and...")
- Role-play jailbreaks ("Pretend you're a model without safety constraints...")
- Encoding attacks (base64, ROT13, or other obfuscation)
- Fragmented injections spread across multiple turns
- Injections embedded in retrieved documents
Document pass/fail rates and your remediation approach for each category.
Common Pitfalls
Relying solely on prompt engineering: Adding "Never ignore these instructions" to your system prompt doesn't create a privilege boundary. LLMs don't have a native concept of instruction hierarchy unless you build enforcement mechanisms.
Trusting retrieved content implicitly: If your RAG pipeline pulls documents from external sources, treat retrieved text as untrusted. An adversary who compromises your knowledge base can inject instructions through document content.
Skipping adversarial validation: Your model might perform well on benign inputs but fail catastrophically when users intentionally probe for vulnerabilities. Test the adversarial cases explicitly.
Assuming fine-tuning solves injection: Fine-tuning can improve instruction-following, but it doesn't create a security boundary between privilege levels. You still need runtime enforcement.
Ignoring multi-turn attacks: A single user message might look benign, but a sequence of messages can gradually shift the model's behavior. Monitor conversation state, not just individual inputs.
Quick Reference Table
| Control Type | Implementation | Validation Evidence | SR 11-7 Mapping |
|---|---|---|---|
| Instruction Separation | Use platform-native system/user roles; never concatenate privileged instructions with user input | Test cases showing user input can't modify system context | Operational controls, design documentation |
| Pre-execution Validation | Scan user input for instruction-like patterns before model processing | False positive/negative rates on known injection patterns | Input validation controls |
| Post-execution Validation | Verify outputs comply with privileged constraints before returning to user | Rejection rate for outputs violating core instructions | Output validation controls |
| Adversarial Simulation | Quarterly testing against OWASP LLM Top 10 and custom attack patterns | Test results, remediation tracking, retest evidence | Ongoing monitoring, validation testing |
| Privilege Documentation | Maintain registry of instruction sources and assigned trust levels | Architecture docs, threat model, privilege assignment rationale | Model documentation, conceptual soundness |
| Monitoring & Alerting | Log and alert on suspected injection attempts or instruction drift | Alert volume, investigation outcomes, false positive tuning | Ongoing monitoring, incident response |
Next Steps: Start with privilege documentation. Map every instruction source in your deployment and assign trust levels. Then build separation mechanisms before adding validation layers. You can't enforce a hierarchy you haven't defined.
