When a federal judge ruled that the Pentagon's actions against Anthropic were "illegal and baseless" after the company sued over its supply chain risk designation, it sent a message beyond one government agency. If your team assesses AI vendors, third-party models, or supply chain risks, your process needs defensible criteria and documented rationale. Otherwise, you're building legal exposure, not managing risk.
This checklist helps you build an AI vendor risk assessment framework that can withstand scrutiny from regulators, auditors, and potentially courts.
What This Checklist Covers
You'll establish clear, legally sound criteria for evaluating AI supply chain risks. This applies whether you're assessing foundation model providers, AI tooling vendors, or data annotation services. The goal is to make transparent, evidence-based decisions that treat vendors fairly while protecting your organization.
This isn't about avoiding all vendor risk. It's about documenting why you made specific risk determinations using defensible methods.
Prerequisites
Before starting this checklist, confirm you have:
- Authority to establish vendor risk criteria: Your legal and procurement teams must approve the assessment framework.
- Access to vendor documentation: Technical documentation, security certifications, model cards, or similar disclosure materials.
- Risk tolerance thresholds: Your organization's defined acceptable risk levels for AI systems.
- Review team composition: Include legal counsel, technical risk assessors, and business stakeholders.
Risk Assessment Framework Checklist
1. Document Your Risk Criteria in Writing
Done when: You've created a written policy defining what constitutes supply chain risk for AI vendors, with specific, measurable criteria.
What good looks like: Your criteria reference concrete factors like security certifications (SOC 2, ISO/IEC 27001), model validation practices, incident response capabilities, or data handling controls. Avoid vague terms like "reputation concerns" or "strategic misalignment" without defining them. If you can't explain the criterion to a judge, don't use it.
2. Establish Evidence Requirements for Each Risk Factor
Done when: For every risk criterion, you've specified what evidence you'll accept as proof of compliance or mitigation.
What good looks like: You've listed acceptable evidence types: third-party audit reports, penetration test results, Responsible Disclosure program documentation, or validated training data provenance records. You've also defined what happens if a vendor can't provide specific evidence (alternative evidence, conditional approval, or disqualification).
3. Create a Tiering System with Clear Thresholds
Done when: You've built a risk tiering framework (low/medium/high or similar) with explicit criteria for each tier.
What good looks like: Your tiers map to specific controls or approval levels. High-risk vendors require executive approval and quarterly reviews. Medium-risk vendors need annual audits. Low-risk vendors follow standard procurement. The assignment logic is documented: "Vendors handling biometric data or providing models for credit decisions automatically qualify as high-risk per EU AI Act Article 6 and SR 11-7 guidance."
4. Build a Vendor Notification Process
Done when: You've defined how and when you'll inform vendors of adverse risk determinations before finalizing them.
What good looks like: Vendors receive written notice of preliminary risk findings with specific deficiencies cited. They get 15-30 business days to respond with corrective evidence or clarification. Your process includes a review mechanism for vendor appeals. Document this in your vendor risk policy.
5. Require Documented Rationale for All Risk Designations
Done when: Every vendor risk assessment includes a written justification citing specific evidence and criteria.
What good looks like: Your assessment template forces reviewers to cite which criteria triggered the risk designation and what evidence supported it. Generic statements like "insufficient security posture" don't pass. Specific findings do: "Vendor failed to provide evidence of annual penetration testing as required by criterion 4.2.b, and incident response plan lacks defined SLAs per criterion 5.1.a."
6. Establish Review Intervals and Triggers
Done when: You've defined when risk assessments get updated and what events force immediate reassessment.
What good looks like: Annual reviews for all vendors, with immediate reassessment triggered by security incidents affecting the vendor, material changes to their AI systems, regulatory enforcement actions, or changes to your own risk tolerance. These triggers are documented in your vendor management policy.
7. Implement Cross-Functional Review for High-Risk Determinations
Done when: High-risk designations require sign-off from legal, technical risk, and business stakeholders before becoming final.
What good looks like: Your approval matrix shows who must review each risk tier. High-risk determinations go through legal counsel review to ensure criteria application is consistent and defensible. Meeting minutes document the review discussion and final decision rationale.
8. Test Criteria Against Hypothetical Scenarios
Done when: You've validated your criteria by applying them to representative vendor scenarios to check for arbitrary or discriminatory outcomes.
What good looks like: Consider a scenario where two vendors offer similar foundation models with identical security certifications but different corporate structures (startup vs. established enterprise). Apply your criteria to both. If one gets flagged as high-risk purely due to company age or size without technical justification, your criteria need refinement.
9. Document Criteria Changes and Grandfather Existing Vendors
Done when: You've created a change management process for updating risk criteria that doesn't arbitrarily penalize existing vendors.
What good looks like: When you add new criteria (like requiring ISO/IEC 42001 certification), you document the effective date, rationale, and transition period. Existing vendors get 6-12 months to meet new requirements. The policy explains why the change was necessary with specific references.
10. Build an Audit Trail
Done when: Every risk assessment, vendor communication, appeal, and decision is retained in a searchable system.
What good looks like: You can reconstruct the complete timeline of any vendor risk determination: initial assessment, evidence reviewed, preliminary findings, vendor response, final decision, and approval chain. Retention period aligns with your record-keeping requirements (typically 7+ years for compliance purposes).
Common Mistakes
- Applying criteria retroactively: Don't designate a vendor as high-risk based on criteria you established after the relationship began without proper notice and transition time.
- Using non-technical factors without justification: Corporate nationality, funding sources, or competitive positioning can't drive risk determinations unless you've documented specific security or compliance implications tied to those factors.
- Skipping the appeal process: Even if you're confident in your assessment, vendors must have a documented path to challenge findings. The absence of this process creates legal vulnerability.
- Treating all AI vendors identically: Your criteria should distinguish between vendors providing general-purpose productivity tools and those supplying models for high-risk use cases. Apply proportionate scrutiny.
Next Steps
After completing this checklist:
- Pilot the framework with 3-5 current vendors across different risk tiers to validate your criteria produce consistent, defensible outcomes.
- Train your procurement and risk teams on applying the criteria and documenting rationale.
- Schedule annual policy review to update criteria based on regulatory changes, emerging threats, or lessons from vendor assessments.
- Coordinate with legal counsel to ensure your vendor risk determinations align with broader compliance obligations under the EU AI Act, SR 11-7, or other applicable frameworks.
The Anthropic case demonstrates what happens when risk designations lack clear criteria and due process. Your AI vendor risk framework should be defensible in court, not just in procurement meetings.
