When Check Point disclosed CVE-2026-35029 and CVE-2026-59823 in LiteLLM, they highlighted a broader issue: your AI infrastructure tools often receive less security scrutiny than your production models. LiteLLM, with nearly 100,000 downloads a month, acts as a bridge between your applications and AI providers like OpenAI, Google, and Anthropic. It manages API keys, logs requests, and enforces access controls. A missing authorization check allowed any authenticated user to gain administrative access, modify configurations, and execute arbitrary code.
This isn't an isolated incident. It's a pattern. As you deploy AI capabilities, the middleware layer is often treated as mere plumbing rather than a potential attack surface. This checklist helps you address that gap.
What This Checklist Covers
This checklist focuses on security controls for AI infrastructure tools, specifically LLM gateways and similar middleware that mediate access between your applications and AI providers. It applies whether you're using LiteLLM, custom gateway code, or commercial alternatives. Each item aligns with a control objective you can verify during internal audits or third-party assessments.
Prerequisites
Before you start:
Inventory your AI infrastructure layer. Know which tools sit between your applications and AI providers, who maintains them, and what credentials they store.
Define roles. Assign someone to own security for AI tooling and another for access management. These roles should be separate.
Test without impacting production. Use a staging environment to verify permission boundaries and simulate attacks.
Checklist Items
1. Verify Current Software Versions
Ensure all LLM gateway instances run patched versions. For LiteLLM, this means version 1.83.9 or later. For other tools, consult vendor security advisories.
Done looks like: A documented inventory showing instance name, version number, and patch date. No instances running versions with known CVEs.
2. Audit Active User Accounts
Review every account with access to your gateway's administrative interface. Remove accounts for former employees, contractors who have left, and unused service accounts.
Done looks like: A user roster with justification for each account, last login timestamp, and sign-off from the account owner's manager. Zero accounts without a clear business need.
3. Test Permission Boundaries
Log in with a low-privilege account. Try to access administrative functions like modifying configurations or viewing all API keys. The gateway should block these actions.
Done looks like: Test results showing that limited roles cannot escalate privileges. Documentation of what each role can and cannot do, verified through actual testing.
4. Rotate Stored API Keys
If you suspect unauthorized access before patching, rotate all AI provider API keys stored in the gateway. This includes keys for OpenAI, Anthropic, Google, and any other providers you integrate with.
Done looks like: New API keys issued by each provider, updated in the gateway, and old keys revoked. A timestamp proving rotation occurred after patching but before returning the gateway to production.
5. Implement Request Logging and Monitoring
Configure comprehensive logging for gateway activity: authentication attempts, configuration changes, API key access, and outbound requests to AI providers. Set up alerts for unusual patterns.
Done looks like: Logs capturing who accessed what, when, from which IP address. Alerts configured for failed authentication attempts, permission changes, and unusual request volumes. Logs retained per your data governance policy.
6. Restrict Network Access
Limit which systems can reach your gateway. Use network segmentation, firewall rules, or cloud security groups to ensure only authorized applications can send requests.
Done looks like: Network diagrams showing gateway placement. Firewall rules explicitly allowing traffic from known application servers and denying all other sources. Verification that direct internet access to the gateway is blocked.
7. Include AI Tooling in Security Assessments
Add LLM gateways and similar infrastructure to your regular penetration testing scope. Treat them with the same rigor you apply to web applications and cloud infrastructure.
Done looks like: Penetration test reports that explicitly cover AI infrastructure. Findings tracked in your vulnerability management system with the same SLA as other critical systems.
8. Establish Responsible Disclosure Channels
If you're running open-source AI tools, monitor security mailing lists and GitHub security advisories. If you've built custom tooling, publish a security policy explaining how researchers should report vulnerabilities.
Done looks like: A documented process for tracking security advisories for every open-source component in your AI stack. For custom tools, a published security.txt file or equivalent disclosure policy.
9. Document Configuration Baselines
Define and document secure configuration standards for your gateway: which features should be disabled, what authentication methods are acceptable, how secrets should be stored.
Done looks like: A configuration baseline document that you can compare against running instances. Automated checks that flag deviations from the baseline.
10. Validate Secrets Management
Confirm that API keys and other credentials aren't hardcoded in configuration files, stored in version control, or logged in plaintext. Use a secrets manager or encrypted storage.
Done looks like: Credentials stored in a dedicated secrets management system (AWS Secrets Manager, HashiCorp Vault, or equivalent). No secrets visible in configuration files, environment variables, or application logs.
Common Mistakes
Assuming gateway security is the vendor's problem. Even with patched software, you're responsible for access controls, network isolation, and monitoring. The vendor fixed the code vulnerability. You need to fix the operational gaps.
Treating AI infrastructure as lower risk than models. Your gateway stores credentials for every AI provider you use. Compromise here means compromise everywhere downstream.
Skipping permission testing. Configuration files often promise role-based access control. CVE-2026-35029 showed the gap between configured permissions and enforced permissions. Test with actual user accounts.
Patching without rotating credentials. If an attacker exploited the vulnerability before you patched, they may have already extracted API keys. Patching closes the door; rotating keys changes the locks.
Next Steps
Start with items 1, 2, and 4. Verify versions, audit accounts, and rotate keys if you have any doubt about prior unauthorized access. These are your immediate containment actions.
Then work through items 3, 6, and 10. Test permissions, restrict network access, and validate secrets management. These prevent the next incident.
Finally, integrate items 5, 7, 8, and 9 into your ongoing security operations. Monitoring, testing, and configuration management aren't one-time tasks. They're continuous controls that should already cover your web applications and cloud infrastructure. Extend them to your AI tooling with the same discipline.
The vulnerabilities Check Point found in LiteLLM are familiar application security failures: a missing permission check, inadequate input validation. What makes them significant is the target. Your LLM gateway centralizes access to AI capabilities across your organization. Secure it accordingly.
