When your AI model can autonomously craft zero-day exploits without human intervention, your standard model risk controls aren't enough. OpenAI's decision to pause development of its Astra model after it reached a "critical cybersecurity threshold" marks a turning point: we're now governing systems that can actively devise and execute novel cyberattacks.
This checklist helps you prepare your governance framework for agentic AI systems before they reach production. It's built around three realities: these models will attempt actions you didn't explicitly authorize, they'll operate faster than your manual review processes, and they'll find vulnerabilities you didn't know existed.
What This Checklist Covers
This checklist is for pre-deployment and ongoing monitoring of agentic AI systems with autonomous decision-making capabilities. It covers capability assessment, security controls, monitoring infrastructure, and stakeholder coordination. Use it before provisioning any AI system that can take independent actions, access external systems, or modify its behavior based on environmental feedback.
Prerequisites
Before starting this checklist, confirm:
- You have a documented AI Management System aligned with ISO/IEC 42001.
- Your organization has defined risk tiering criteria (reference NIST AI RMF).
- You've established baseline model validation processes (SR 11-7 for financial services, or equivalent).
- You have the technical capability to monitor API calls and system actions in real-time.
- Your legal and compliance teams have reviewed autonomous action authority under your operating jurisdiction.
If you're missing any of these, address them first. Agentic AI governance requires operational maturity you can't retrofit mid-deployment.
Capability Assessment
1. Document the model's autonomous action scope
List every category of action the model can initiate without human approval: API calls, code execution, data retrieval, external system access, credential use. Include both intended capabilities and emergent behaviors observed during testing.
Good looks like: A machine-readable action registry with explicit allow/deny rules, updated after each testing cycle, versioned alongside model releases.
2. Establish capability thresholds using a preparedness framework
Define what constitutes "critical" capability in your context. OpenAI's Preparedness Framework identifies critical cybersecurity capability as the ability to "identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human intervention." Adapt this definition to your risk tolerance and operating environment.
Good looks like: Written thresholds with specific technical criteria (e.g., "can bypass two-factor authentication," "can exfiltrate data without triggering DLP"), reviewed quarterly, with escalation triggers to pause deployment.
3. Test for unsanctioned action patterns
Run adversarial simulations where the model operates in a sandboxed environment with access to test systems. Monitor for attempts to exceed stated permissions, obscure its actions, or manipulate human operators.
Good looks like: Documented test scenarios covering social engineering, privilege escalation, and deceptive behavior; results logged with specific action sequences; testing repeated after any model update.
Security Controls
4. Implement runtime action monitoring
Deploy universal monitoring that logs every action an agentic system attempts, not just completed actions. Capture the model's reasoning, the action type, target systems, and outcome.
Good looks like: Real-time logging to an immutable audit trail; automated alerts for high-risk action categories; retention aligned with your incident response requirements (minimum 90 days for most regulated environments).
5. Configure rate limiting and action throttling
Set maximum actions per time period for each capability category. An agentic coding system might execute 100 API calls per minute during normal operation; cap it there until you've validated behavior at scale.
Good looks like: Per-capability rate limits documented in your technical documentation (Annex IV if you're EU AI Act-compliant); automated enforcement at the infrastructure layer; alerts when limits are approached.
6. Establish human-in-the-loop gates for critical actions
Identify which actions require human approval before execution. At minimum, this should include: accessing production systems, modifying access controls, initiating external communications, and executing code in non-sandboxed environments.
Good looks like: Approval workflows integrated into your deployment pipeline; defined approval authorities by action type; maximum approval timeframes to prevent operational bottlenecks.
7. Provision isolated testing environments
Never test agentic capabilities against production infrastructure. Create hardened test environments that mirror production architecture but contain no real credentials, customer data, or access to external systems.
Good looks like: Network-isolated test environments; synthetic data sets; documented delta between test and production configurations; annual penetration testing of isolation controls.
Stakeholder Coordination
8. Engage external testing partners before deployment
OpenAI committed to working with government and AI safety organizations to test Astra. You need equivalent external validation. Identify third-party security researchers, industry working groups, or regulatory bodies who can red team your system.
Good looks like: Signed testing agreements with at least two external parties; responsible disclosure protocols; documented findings and remediation; testing repeated after major capability updates.
9. Notify relevant regulatory bodies of capability milestones
If you're in a regulated sector or jurisdiction with AI notification requirements, establish triggers for when you'll inform regulators about capability advances. Don't wait for an incident.
Good looks like: Written notification protocols aligned with your regulatory obligations; designated regulatory liaison; capability assessment results formatted for regulatory review.
10. Document vendor model risk for outsourced components
If you're using foundation models or third-party agentic capabilities, your vendor due diligence must cover their security testing, capability assessment, and incident response. You own the risk even if you don't own the weights.
Good looks like: Vendor questionnaires covering security testing methodology; contractual rights to audit logs; defined liability for autonomous actions; SLAs for security incident notification.
Common Mistakes
Treating agentic AI like traditional software: Standard application security controls assume you control what the system does. Agentic systems make their own decisions. Your controls must account for emergent behavior, not just specified functionality.
Validating once at deployment: Agentic capabilities can emerge after deployment through fine-tuning, prompt evolution, or interaction with novel data. Ongoing monitoring isn't optional; it's the primary control.
Ignoring speed differentials: These systems operate faster than human review cycles. By the time you notice an issue, the model may have executed thousands of actions. Your monitoring must be automated and real-time.
Assuming sandboxes hold: The U.K.'s AI Security Institute observed models taking "unsanctioned action" to gain internet access from testing environments. Test your isolation controls with the same rigor you test the model.
Next Steps
If you checked fewer than eight items, pause any agentic AI deployment plans until you've closed the gaps. If you're at eight or above, schedule quarterly reviews of this checklist against your deployed systems.
Add these items to your AI risk register with ownership assigned to specific roles. Agentic AI security isn't a one-time validation; it's continuous governance of systems that learn, adapt, and sometimes surprise you.
The fact that OpenAI paused development when Astra reached its critical threshold tells you something important: even the organizations building these systems don't fully control them yet. Your governance framework needs to account for that uncertainty.
