If you're responsible for securing AI systems in hiring, credit, healthcare, or critical infrastructure, you're now under a regime that treats model failures like operational incidents. This guide breaks down what the EU AI Act requires when your AI system is classified as high-risk.
What This Guide Covers
This guide addresses the technical control requirements for high-risk AI systems under the EU AI Act. It's for security engineers, platform teams, and technical compliance leads who need to implement, audit, or maintain these controls in production environments.
You'll find requirement breakdowns, implementation patterns, and a quick-reference table. This guide does not cover prohibited practices, minimal-risk systems, or General-Purpose AI Model provider obligations in detail.
Key Concepts and Definitions
High-Risk AI System: An AI system that influences decisions affecting a person's rights, safety, livelihood, or access to essential services. Examples include employment screening tools, credit-scoring models, biometric identification systems, and AI in medical devices or critical infrastructure.
Risk Classification: Determined by intended purpose and context of use, not model architecture. A foundation model becomes high-risk when connected to an HR system with candidate-ranking authority but remains lower-risk when summarizing internal meeting notes.
Provider vs. Deployer: The provider develops or modifies the AI system and places it on the market. The deployer uses it under their own authority. Your obligations differ based on your role, and integrating a foundation model into a recruitment platform can shift you into provider territory.
System Boundary: The documented scope that defines what model is used, what data sources feed it, what tools it can call, what decisions it influences, and whether human review exists. This boundary determines risk classification and must be reassessed when any component changes.
Requirements Breakdown
1. Risk Management Process
You need a documented process that identifies:
- Intended purpose and foreseeable misuse scenarios
- Affected populations and potential harms
- Mitigation decisions with rationale
- Residual risk acceptance criteria
This isn't a one-time document. When you change the model version, add a retrieval layer, or expand the user population, you're changing the risk profile.
2. Data Governance
Your control environment must address:
- Lineage: Can you trace training, validation, and test datasets back to their sources?
- Bias analysis: Have you tested for representation gaps or performance disparities across demographic groups?
- Privacy controls: Are Data Protection Impact Assessments current, and do they reflect how the model processes personal data?
3. Technical Documentation and Traceability (Annex IV)
You must be able to reconstruct what produced a given output. That means capturing:
- Model version and configuration at inference time
- Prompt or input that triggered the decision
- Retrieval sources if using RAG architecture
- Tool calls if the system has agentic capabilities
- Human override events
If your logging infrastructure can't answer "What model version, with what prompt, using what context, produced this output on this date?", you don't have traceability.
4. Meaningful Human Oversight
A person clicking "approve" after the AI has already shaped the decision isn't real oversight. The human needs:
- Sufficient information to understand the AI's reasoning
- Authority to challenge or override the system
- Enough time to review before the decision executes
- Clear accountability if they approve a harmful output
5. Accuracy, Robustness, and Cybersecurity
Accuracy means performance metrics tied to the intended purpose, not just benchmark scores. If your model is used for candidate ranking, you need metrics that reflect ranking quality, not just text-generation perplexity.
Robustness Testing must include adversarial scenarios: prompt injection attempts, out-of-distribution inputs, retrieval poisoning, and edge cases where the model's probabilistic behavior could produce harmful outputs.
Cybersecurity controls span model versioning, access management, secrets handling, API security, and monitoring for model inversion or data extraction attacks.
6. Post-Market Monitoring and Incident Reporting
You're required to monitor production behavior and report serious incidents. This means:
- Detecting when model outputs drift from expected behavior
- Tracking user complaints or override patterns
- Identifying when accuracy degrades for specific subpopulations
- Reporting incidents that result in serious harm or fundamental rights violations
7. Change Control
When you swap the foundation model, update system instructions, add a new tool, or modify the retrieval layer, you're potentially changing the system's risk classification. Your change process must re-evaluate controls, update documentation, and confirm that transparency obligations still function.
Implementation Guidance
Start With System Inventory
Before you can implement controls, you need to know what you're controlling. Build an inventory that captures:
- Every AI system in scope (including chatbots, recommendation engines, automated decision tools)
- Intended purpose and decision authority
- Model provider and version
- Data sources (training, retrieval, user inputs)
- Integration points and downstream systems
- Current risk classification
Implement Logging Early
Traceability requirements mean you can't retrofit logging after the fact. Instrument your systems to capture:
- Model inputs and outputs (respecting privacy controls)
- Retrieval queries and sources
- Tool execution events
- Human review decisions
- System errors and fallback triggers
Design for Human Oversight
If your system requires human oversight, design the interface to support real review:
- Surface the AI's confidence level and reasoning
- Show alternative options the model considered
- Provide context the human needs to challenge the output
- Log when humans override and why
Test Transparency Controls
For Article 50 compliance (effective August 2, 2026), test whether:
- Disclosures appear before or at the point of interaction
- Generated content carries appropriate metadata
- Labels persist after export, compression, or republishing
- APIs preserve provenance information
- Downstream systems can't strip markings without detection
Common Pitfalls
Treating risk classification as static: Your model's risk tier can change when you modify its purpose, expand its decision authority, or integrate it into a new workflow. Classification is tied to system boundaries, not model architecture.
Logging inputs but not context: Capturing the user's prompt isn't enough. You need the full context window, retrieval results, system instructions, and tool state to reconstruct what influenced the output.
Assuming the foundation model provider handles everything: If you're integrating a model into a high-risk application, you're likely a provider under the Act, which means you inherit the full control stack regardless of who built the underlying model.
Implementing disclosure as an afterthought: If your transparency control is a label added at the UI layer, it won't survive API calls, exports, or content transformations. Design for provenance from the start.
Skipping adversarial testing: Benchmark performance doesn't predict robustness under attack. You need to test prompt injection, retrieval manipulation, and jailbreak attempts specific to your system's architecture.
Quick Reference Table
| Requirement | Key Artifacts | Update Trigger | Owner |
|---|---|---|---|
| Risk Management | Risk assessment, mitigation plan, residual risk acceptance | Purpose change, population expansion, new integration | Product + Security |
| Data Governance | Dataset lineage, bias analysis, Data Protection Impact Assessment | New data source, model retrain, population shift | Data + ML Ops |
| Technical Documentation (Annex IV) | System design doc, model card, integration spec | Model swap, architecture change, tool addition | Engineering + Compliance |
| Traceability | Inference logs, retrieval logs, tool execution logs, override events | Logging schema change, retention policy update | Platform + Security |
| Human Oversight | Review interface, override workflow, escalation path | Decision authority change, automation level increase | Product + Legal |
| Accuracy & Robustness | Performance metrics, adversarial test results, drift detection | Model version change, performance degradation | ML Ops + Security |
| Cybersecurity | Threat model, access controls, API security, secrets management | New integration, privilege change, external exposure | Security + Infrastructure |
| Post-Market Monitoring | Incident log, user complaints, override patterns, drift alerts | Serious incident, regulatory query, performance anomaly | ML Ops + Compliance |
| Change Control | Change request, risk reassessment, control validation | Any system modification | Engineering + Compliance |
| Transparency (Article 50) | Disclosure implementation, metadata schema, provenance testing | Content generation feature, new output channel | Product + Legal |
Penalties for non-compliance: The highest penalty tier can reach €35 million or 7% of worldwide annual turnover. Operational consequences include deployment restrictions, market withdrawal requirements, and customer contract terminations before any fine is calculated.
Next deadline: August 2, 2026, for Article 50 transparency obligations. If your system generates or manipulates content, or if users interact with AI in ways requiring disclosure, you need controls in place by that date.
