When a nonprofit showed that jailbreaking frontier AI models costs less than a tank of gas, many compliance teams missed the point. They're still debating whether their existing model validation frameworks cover adversarial attacks, operating under dangerous misconceptions about model security.
These myths don't just create blind spots. They prevent you from building controls that catch real vulnerabilities before deployment. Let's clear them up.
Myth 1: "Our vendor's safety testing is sufficient"
Reality: FAR.AI found 448 jailbreaks in SpaceXAI's Grok models and 249 in Google's Gemini models using automated prompt generation. The cost to compromise Grok was $58, and for Gemini, $278.
Your vendor's internal red teaming catches what they're looking for. It doesn't catch what external actors with different threat models will try. When FAR.AI tested four major model providers using the same adversarial methodology, results varied wildly. Some models showed no successful jailbreaks under their test conditions; others were extensively compromised.
The compliance implication: Vendor testing reports aren't validation evidence. They're marketing documents until you verify them independently. Your SR 11-7 effective challenge requires you to test claims, not accept them. This means running your own adversarial simulations against use-case-specific risks, not generic "harmful content" categories.
Myth 2: "If the model refuses most harmful prompts, it's safe"
Reality: FAR.AI's tool generated over a thousand prompt variations for each test case. Models rejected many attempts outright before a jailbreak succeeded. That's how adversarial attacks work. The attacker only needs one working exploit.
You're not measuring safety by refusal rate. You're measuring it by whether any prompt variation succeeds at bypassing controls. A model that refuses 999 attempts but generates detailed cyberattack plans on the thousandth try isn't "99.9% safe." It's compromised.
This matters for your risk tiering. A high-risk AI system under the EU AI Act or a model subject to SR 11-7's heightened validation requirements can't rely on "usually works" guardrails. Your Technical Documentation (Annex IV) needs to demonstrate systematic robustness testing, not spot checks.
Myth 3: "We'll catch problems in Post-Market Monitoring"
Reality: Adversarial attacks are designed to look benign until they're not. By the time your Post-Market Monitoring flags an issue, the model has already generated the exploit code, the misinformation campaign, or the biased decision.
Consider what monitoring would need to catch a jailbreak in production: You'd need to recognize that a seemingly innocuous prompt about "creative writing" or "educational research" was actually engineered to bypass safety filters. That requires understanding attacker intent, not just output content.
Your monitoring controls should assume jailbreaks will be attempted and focus on limiting blast radius, not preventing all attacks. Implement rate limiting by user, output filtering, human review for high-stakes decisions, and automatic session termination when anomalous patterns emerge. Model Limitations and Use Restrictions should explicitly define what happens when the model behaves unexpectedly, not just when it performs poorly.
Myth 4: "Regulation will solve this"
Reality: California and New York now require frontier AI developers to publish safety reports. Illinois will soon mandate third-party audits of safety practices. None of these laws specify what "safe enough" means or create liability for failures.
Adam Gleave from FAR.AI argues that "AI models right now are less regulated than restaurants." He's right about the regulatory gap, but wrong if you interpret that to mean compliance teams should wait for clear standards before acting.
Your AI Management System under ISO/IEC 42001 doesn't wait for perfect regulatory clarity. It establishes controls based on identified risks, documents your risk treatment decisions, and creates audit trails showing you took reasonable precautions. When regulations do arrive, you'll have validation evidence demonstrating you were already managing the risks they formalize.
The EU AI Act's General-Purpose AI Code of Practice will eventually specify systemic risk evaluation methods. Until then, document why your current testing regime is proportionate to deployment risk and what additional controls you'd implement if threat intelligence changes.
Myth 5: "This is a technical problem for the AI team"
Reality: Jailbreak vulnerability is a vendor due diligence issue, a contractual liability question, and a disclosure obligation under multiple frameworks.
When you deploy a Foundation Model Provider's API, you're accepting their security posture. If that model can be jailbroken for under $100, what's your liability when someone uses your application to generate harmful content? Your vendor contract probably doesn't specify. Your AI System Impact Assessment under ISO/IEC 42005 should.
Legal and compliance teams need to ask:
- What indemnification does our contract provide for model safety failures?
- Do we have Responsible Disclosure protocols when we discover jailbreaks?
- Are we required to report model security incidents under sector-specific regulations?
- What constitutes "reasonable security measures" for AI systems in our jurisdiction?
These aren't technical questions. They're risk transfer and regulatory interpretation questions that most AI teams aren't equipped to answer.
What to do instead
Stop treating model security as a binary state. You're not building "safe" or "unsafe" systems. You're managing attack surface and limiting damage.
Start with threat modeling. Who would want to jailbreak your specific deployment, and what would they gain? A customer service chatbot and a code generation tool face different adversaries with different capabilities.
Then build layered controls: input validation, output filtering, rate limiting, human review for high-stakes outputs, and automated circuit breakers. Document these as Annex A Controls in your AI Management System, not as optional practices.
Finally, test your assumptions. Run your own adversarial simulations. Hire external red teams. Assume your vendor's safety testing missed something, because FAR.AI just demonstrated it probably did.
The models are already deployed. The jailbreaks are already cheap. Your job isn't to wait for perfect safety. It's to build controls that limit what happens when safety fails.
