Skip to main content
Commerce Security logo, "All 12 PCI DSS Requirements in Plain English," "Get it now for free," "Complete Survival Guide" and a button toclick to get it
Vetting Your External AI Evaluators Under the EU AI ActEU AI Act & GPAI
3 min readFor AI Governance Leaders

Vetting Your External AI Evaluators Under the EU AI Act

External evaluation of General-Purpose AI Models with systemic risk is becoming a mandatory practice. Article 55(1)(b) of the EU AI Act and the GPAI Code of Practice emphasize the importance of external evaluation in assessing and mitigating systemic risk. On July 15, 2026, the AI Office proposed requirements for the qualifications and independence of external evaluators at a workshop with selected organizations. They also plan to develop a scoring system for evaluators.

While these proposals are not yet finalized, this checklist provides guidance on selecting, contracting, and documenting evaluators in a way that aligns with the anticipated requirements. It's designed for AI governance leaders who need to ensure accountability and transparency in AI model assessments.

Prerequisites

Before proceeding, ensure you have:

Checklist

1. Map your evaluation obligations to Article 55(1)(b).
Document which models require external evaluation, the evaluation activities committed to under the GPAI Code of Practice, and ownership of each. A clear obligation map should be understandable to a regulator without explanation.

2. Define evaluator qualification criteria before shortlisting.
Evaluators need domain expertise and evaluation skills. Develop a rubric that includes relevant domain knowledge, evaluation methodology, and secure testing capability. This rubric should be approved by governance before engaging vendors.

3. Test independence across three dimensions.
Ensure evaluators are free from your managerial, financial, or operational influence. Check if anyone on your team directs the evaluator's work, assess the financial relationship, and confirm they don't rely on your infrastructure for testing.

4. Run and refresh a conflict-of-interest declaration.
Gather disclosures on prior consulting work, equity, advisory roles, and shared personnel. Update these declarations at renewal or when the engagement scope changes.

5. Lock evaluator discretion over scope and publication of findings.
Ensure contract language prevents your team from altering conclusions or narrowing the test scope. Evaluators should have the authority to report their findings without edits.

6. Plan the model access mechanism.
Specify the access level (API, weights, internal tooling), environment, and timeline. Budget for secured testing environments rather than assuming the evaluator will cover these costs.

7. Scope evaluation beyond cybersecurity.
Include risks like loss of control over models and AI manipulation of users. Ensure your evaluation plan addresses each systemic risk in your register.

8. Define how findings enter your risk process.
Set severity thresholds, remediation owners, and deadlines. Decide which findings should halt deployment.

9. Connect findings to serious incident reporting.
Article 55(1)(c) requires reporting serious incidents without delay. Plan how to escalate and assess reportability of evaluator discoveries.

10. Prepare for information requests from authorities.
Under Article 77, authorities can request documents created under the Act. Store evaluator reports in a retrievable, access-controlled repository.

11. Check your authorized representative position.
If you're outside the EU, appoint an authorized representative in the Union who knows where evaluation records are kept.

12. Review the evaluator relationship regularly.
Incorporate reassessment of evaluators into your ISO/IEC 42001 PDCA loop. Conduct annual re-scoring against your rubric and consider rotation or retender policies.

Common Mistakes

  • Assuming reputation equals independence. Reputation doesn't guarantee independence from managerial, financial, or operational influence.
  • Paying evaluators based on outcomes. Avoid outcome-contingent fees that compromise control measures.
  • Limiting scope to easy tests. Don't neglect harder tests like loss-of-control and manipulation.
  • Assuming the Code of Practice is optional. While technically voluntary, it's the main compliance route until harmonized standards are established.
  • Misplacing evaluator reports. Ensure compliance can quickly retrieve reports to avoid audit issues.

Next Steps

Begin with items 2 and 3. A written rubric and independence test are inexpensive and prompt essential discussions early. Monitor the AI Office's evolving evaluator requirements and compare them with your rubric. Document any discrepancies. Use the AI Act Service Desk for interpretive questions and keep records of these exchanges.

Finally, assign someone to oversee evaluation capacity planning. The Commission aims to establish an EU evaluation capacity by 2027, which could impact who evaluates your models and how.

Promotional banner for the Penetration Report Template Kit

You Might Also Like