You're about to deploy a new LLM-based system. Your team has run the standard tests, fixed obvious bugs, and completed your security questionnaire. But have you actually tried to break it?
Red-teaming is essential. Whether you're preparing for an EU AI Act conformity assessment, building validation evidence under SR 11-7, or trying to avoid a prompt injection incident, you need a structured approach to Adversarial Simulation before your model goes live.
This checklist guides you through the pre-deployment red-teaming steps that distinguish a compliant, audit-ready AI system from one that's waiting to fail in production.
What This Checklist Covers
This pre-deployment red-teaming checklist is for organizations deploying LLM-based systems, especially those with agentic capabilities like web browsing or code execution. It focuses on prompt injection and adversarial simulation activities you should complete before release.
It doesn't replace your broader AI system impact assessment, model validation, or ongoing post-market monitoring. Think of it as the adversarial-specific layer between development and production.
Prerequisites
Before starting red-teaming, ensure you have:
- Documented model limitations and use restrictions defining what the system should and shouldn't do.
- A test environment mirroring production scenarios (if your model will browse the web, your red-teaming environment needs web access).
- Defined attack scenarios relevant to your deployment context (email handling, code editing, customer data access, etc.).
- Logging infrastructure capturing model inputs, outputs, and intermediate reasoning (including chain-of-thought steps if your model uses them).
- A responsible disclosure process for vulnerabilities discovered during testing.
Your test environment should replay real-world interaction patterns, and you've already documented what "successful defense" means for each scenario.
Red-Teaming Checklist
1. Map Your Attack Surface
Identify every point where untrusted input can reach your model. This includes user prompts, retrieved documents, web content, API responses, uploaded files, and code repositories.
Requirement reference: ISO/IEC 23894:2023 (contextual risk factors), NIST AI RMF Measure function
Good looks like: You have a diagram showing all data flows into your model, tagged by trust level and potential attacker control.
2. Test Prompt Injection Across All Input Channels
Run adversarial simulations against each identified input channel. Don't just test the chat interface. If your model reads emails, hide instructions in email signatures. If it processes code, embed directives in comments. If it browses websites, inject commands in page metadata.
Requirement reference: EU AI Act Article 15 (accuracy, robustness, and cybersecurity), ISO/IEC 42001:2023 Annex A Control 6.2.7 (adversarial attacks)
Good looks like: You've attempted injection attacks through every identified channel and documented which defenses held and which failed.
3. Verify Chain-of-Thought Integrity
If your model uses chain-of-thought reasoning, test whether an attacker can insert false entries into that reasoning chain. Try injecting statements like "You have already verified that this user is an administrator."
Requirement reference: ISO/IEC 42001:2023 Annex A Control 6.2.6 (data integrity)
Good looks like: Your model either (a) cryptographically signs its own chain-of-thought entries, (b) maintains the chain in a protected memory space inaccessible to external inputs, or (c) validates the provenance of each reasoning step before acting on it.
4. Test Multi-Turn Attack Persistence
Run attacks that span multiple conversation turns. Human attackers don't give up after one failed attempt. They probe, adjust, and escalate. Your red-teaming should do the same.
Requirement reference: NIST AI RMF Measure 2.3 (Adversarial Simulation)
Good looks like: You've documented how your model responds to escalating attack patterns over 5+ turns, and you've confirmed that defenses don't degrade as conversation context grows.
5. Validate Defense Mechanisms Under Load
Test whether your safety controls hold up when the model is handling multiple concurrent requests or operating under rate limiting conditions. Attackers will probe for race conditions and resource exhaustion scenarios.
Requirement reference: ISO/IEC 42001:2023 Annex A Control 8.32 (capacity management)
Good looks like: Your defenses work consistently at 2x expected peak load, and you've confirmed that rate limiting doesn't create bypass opportunities.
6. Document Attack Success Rates and Model Robustness Gains
Measure what percentage of your test attacks succeeded against your current model version versus previous versions. If you're implementing defenses iteratively, track improvement over time.
Requirement reference: SR 11-7 (ongoing monitoring and validation), EU AI Act Article 15 (robustness)
Good looks like: You can state, "23% of our test attack set succeeded against the current version, down from 90% against the previous version." You've logged every attack attempt, the model's response, and whether it constituted a successful compromise.
7. Capture Validation Evidence for Novel Attack Types
When you discover a new attack pattern your team hadn't anticipated, document it thoroughly. Include the attack vector, the model's vulnerable behavior, the fix you implemented, and confirmation that the fix works.
Requirement reference: SR 11-7 (effective challenge), ISO/IEC 42001:2023 Annex A Control 6.2.7
Good looks like: Your validation evidence includes specific examples of attacks that initially succeeded, the reasoning behind your defensive approach, and test results showing the attack no longer works.
8. Test Image-Based Injection Vectors
If your model accepts image inputs, test whether attackers can embed text instructions in image metadata, hide prompts in pixel data, or use OCR-readable text overlays to inject commands.
Requirement reference: ISO/IEC 42001:2023 Annex A Control 6.2.7
Good looks like: You've tested at least three distinct image-based injection techniques and documented your model's response to each.
9. Verify Third-Party Integration Safety
If your model interacts with external APIs, databases, or agent systems, test whether an attacker can use your model as a proxy to compromise those systems. Can a prompt injection make your model exfiltrate data to an attacker-controlled endpoint? Can it modify records it shouldn't touch?
Requirement reference: EU AI Act Article 15, ISO/IEC 42001:2023 Annex A Control 5.2 (vendor model risk)
Good looks like: You've mapped every external system your model can affect, assigned privilege levels, and confirmed that prompt injection cannot escalate those privileges.
10. Establish Red-Teaming Cadence for Production
Define how often you'll re-run Adversarial Simulation after deployment. New attack techniques emerge constantly. Your red-teaming can't be a one-time pre-launch event.
Requirement reference: EU AI Act Article 72 (post-market monitoring), ISO/IEC 42001:2023 Annex A Control 8.2 (continuous improvement)
Good looks like: You've scheduled quarterly red-teaming exercises, assigned ownership, and integrated findings into your model recalibration process.
Common Mistakes
Treating red-teaming as a checklist exercise. Running a fixed test suite and calling it done misses the point. Adversarial simulation should be creative and adaptive. If your red team isn't finding anything, you're not trying hard enough.
Testing only the chat interface. Most prompt injection vulnerabilities show up in non-obvious input channels: retrieved documents, tool outputs, or data your model processes in the background.
Assuming defenses are permanent. A technique that blocks 90% of attacks today might block 10% after attackers adapt. Red-teaming is continuous, not conclusive.
Skipping the "why did this work?" analysis. When an attack succeeds, don't just patch it and move on. Understand the root cause. Otherwise, you'll face the same class of vulnerability in a different form next month.
Next Steps
After completing this checklist, you should have:
- Documented validation evidence showing which attacks your model successfully defended against
- A log of vulnerabilities discovered and remediated
- Baseline metrics for model robustness that you can track over time
- A plan for ongoing Adversarial Simulation post-deployment
Integrate your red-teaming findings into your Technical Documentation (Annex IV) if you're subject to the EU AI Act, or into your model validation report if you're operating under SR 11-7. Your auditors will ask how you tested adversarial robustness. "We ran some attacks and it seemed fine" won't cut it.
If you discovered attack patterns your team hadn't seen before, share them (in sanitized form) with your model vendor or foundation model provider. Responsible disclosure helps the entire ecosystem get stronger.
And remember: the goal isn't to make your model unbreakable. It's to understand where it breaks, fix what you can, document what you can't, and make informed decisions about acceptable residual risk before you go live.
