You're managing AI risk where institutional controls lag behind deployment speed. Recent incidents with OpenAI and Hugging Face, along with ongoing PHP vulnerabilities, aren't isolated failures. They're symptoms of a structural problem: competitive pressure is outpacing your ability to govern what you build and deploy.
This checklist helps you assess whether your AI safety program addresses threats across three critical dimensions: technological controls, legal preparedness, and regulatory compliance. Each dimension requires different expertise, timelines, and stakeholders. All three must function simultaneously.
Prerequisites
Before starting this assessment:
- Inventory your AI systems. You can't protect what you haven't cataloged. Include internal models, third-party APIs, and systems with AI components from open-source repositories.
- Map your threat surface. Identify where competitive pressure (internal or external) creates shortcuts in your validation, testing, or deployment processes.
- Assign ownership. Technological controls are managed by engineering and data science. Legal mechanisms require counsel and contracts teams. Regulatory compliance sits with your governance office. All three need a coordination point.
Safety Program Checklist
Technological Controls
1. Supply chain verification for AI components
□ Maintain an inventory of all AI models, libraries, and dependencies used in production systems
□ Verify the provenance and integrity of models from external repositories (including Hugging Face, GitHub, or internal model registries)
□ Scan dependencies for known vulnerabilities before integration
□ Define acceptable risk thresholds for third-party AI components
What good looks like: Your team can trace every model in production back to its source, knows what training data it touched, and has evidence that it passed security scanning before deployment.
2. Red teaming for adversarial threats
□ Conduct Red Teaming exercises that simulate adversarial attacks on your AI systems
□ Cover prompt injection, data poisoning, model extraction, and evasion attacks in your testing
□ Document Red Teaming findings and track remediation
□ Incorporate MITRE ATLAS threat patterns into your test scenarios
What good looks like: You can show auditors a testing schedule, attack scenarios you've run, vulnerabilities you found, and how you closed them.
3. Robustness testing under distribution shift
□ Test model behavior when input distributions differ from training data
□ Define performance thresholds that trigger revalidation
□ Monitor for concept drift in production
□ Document a process to retrain or retire models that degrade
What good looks like: Your monitoring detects when a model's accuracy drops below acceptable levels, and you have a runbook for response.
4. Secure development practices for AI systems
□ Follow ISO/IEC 5338 lifecycle processes or equivalent for AI development
□ Require code review before merging changes to production models
□ Separate training environments from production environments
□ Log and restrict access to production model weights and training data
What good looks like: An unauthorized engineer can't push an untested model to production, and you can prove it with access logs and deployment controls.
Legal Mechanisms
5. Contractual protections for AI services
□ Specify model performance standards, data handling requirements, and liability allocation in vendor contracts
□ Include Responsible Disclosure terms for security vulnerabilities
□ Negotiate indemnification for IP claims related to model outputs
□ Define acceptable downtime and degradation thresholds in service-level agreements
What good looks like: If a vendor's model fails or produces harmful outputs, your contract defines who's liable and what remedies you have.
6. Intellectual property and data rights clarity
□ Document ownership of training data, model weights, and fine-tuned derivatives
□ Review and approve licensing terms for open-source models before use
□ Assign AI-related IP to your organization in employee and contractor agreements
□ Assess whether your models' training data includes copyrighted material
What good looks like: You can answer "who owns this model?" and "can we use this data?" without legal ambiguity.
7. Liability framework for AI-caused harm
□ Identify high-risk use cases where AI errors could cause physical, financial, or reputational harm
□ Ensure insurance coverage addresses AI-specific risks (not just general tech E&O)
□ Include legal notification requirements in incident response plans
□ Map which laws apply to your AI systems (product liability, consumer protection, sector-specific rules)
What good looks like: When an AI system causes harm, you know which legal regime applies, who gets notified, and how your insurance responds.
Regulatory Compliance
8. Risk tiering under applicable frameworks
□ Classify your AI systems by risk level using EU AI Act criteria, NIST AI RMF, or equivalent
□ Ensure high-risk systems have enhanced documentation, testing, and oversight
□ Identify any systems that might qualify as Prohibited AI Practices
□ Review risk classifications when systems change or regulations update
What good looks like: You can show regulators a risk inventory with clear justifications for each tier.
9. Technical documentation (Annex IV) for high-risk systems
□ Ensure high-risk AI systems have documentation covering intended purpose, data requirements, model architecture, and performance metrics
□ Include known limitations and failure modes in documentation
□ Maintain records of training data characteristics (even if you can't share the data itself)
□ Version-control and update documentation with each model release
What good looks like: An external auditor can understand what your model does, how it was built, and what could go wrong.
10. Post-market monitoring and incident reporting
□ Track AI system performance in production and log anomalies
□ Define thresholds that trigger incident investigation
□ Report serious incidents to relevant authorities within required timeframes
□ Maintain records of incidents, root causes, and corrective actions
What good looks like: You detect problems before users do, and regulators see evidence of active monitoring, not just compliance theater.
11. Governance structure with accountability
□ Assign an AI governance owner with authority to halt deployments
□ Ensure board or executive leadership receives regular AI risk reports
□ Have cross-functional teams (legal, engineering, compliance) meet to review AI risks
□ Document escalation paths for safety concerns
What good looks like: Engineers know who to call when they spot a safety issue, and that person can stop a launch.
Common Mistakes
Treating this as a technology-only problem. Your best Red Teaming won't help if your vendor contract leaves you liable for their model's failures. Your compliance documentation won't matter if you haven't tested for adversarial attacks.
Waiting for regulatory clarity. The EU AI Act exists. NIST AI RMF exists. SR 11-7 applies if you're a financial institution. You don't need perfect regulation to start building controls.
Assuming competitive pressure justifies shortcuts. The global race with China and internal competition among U.S. AI labs create pressure to ship fast. That pressure doesn't exempt you from liability when your system fails.
Siloing the three dimensions. Technology, legal, and regulatory teams need a shared view of your AI risk landscape. If they're working in parallel without coordination, gaps will emerge.
Next Steps
- Run this checklist with representatives from engineering, legal, and compliance in the room. Note where you have gaps.
- Prioritize based on your risk tier. High-risk systems need all eleven items. Lower-risk systems can phase in controls.
- Set a 90-day review cycle. AI safety isn't a one-time audit. Competitive dynamics, new vulnerabilities, and regulatory changes require continuous reassessment.
- Document your gaps and your plan. If an incident happens before you've closed every gap, you'll need to show you were working on it systematically, not ignoring known risks.
The AI safety crisis isn't hypothetical. It's playing out in lawsuits, security incidents, and regulatory enforcement. Your program needs to address all three dimensions simultaneously, or you're leaving yourself exposed.
