Is Your Therapy Chatbot Banned in the EU?

If your team is deploying AI systems for users seeking emotional support, you're in a zone the EU AI Act scrutinizes closely. The regulation doesn't just add compliance tasks, it clearly defines what you can and cannot deploy, effective immediately for prohibited systems.

This checklist guides you through compliance obligations for AI systems used in therapy or emotional support. Whether you're building a mental health app or a general-purpose system that users might turn to for support, these requirements apply to you.

What This Checklist Covers

This checklist addresses three compliance layers under the EU AI Act:

1. Prohibited practice screening (Article 5(1)(b))
2. High-risk system obligations (if your system qualifies as a medical device)
3. Transparency requirements (Article 50)

It applies to both narrow therapeutic AI systems and General-Purpose AI Models like ChatGPT or Claude when used for therapy or emotional support. The distinction matters because your obligations exist at both the system and model levels if you're integrating your own model.

Prerequisites

Before starting this checklist, confirm:

• You've identified all AI systems that could be used for therapy, emotional support, or mental health purposes (including unintended uses).
• You know whether you're the provider placing the system on the EU market, putting it into service, or both.
• You've documented the system's intended purpose and have access to all promotional materials, instructions for use, and user-facing documentation.
• You understand whether your system integrates a third-party model or your own model (different compliance paths apply).

Compliance Checklist

Prohibited Practice Screening

1. Vulnerability exploitation assessment

Evaluate whether your system exploits vulnerabilities related to age, disability, or specific social/economic situations with the objective or effect of materially distorting behavior likely to cause significant harm.

• Review system design for features that could exploit limited intellectual capacity, emotional distress, or age-related vulnerabilities.
• Analyze whether the system nudges users toward harmful behaviors or expensive purchases.
• Document reasonably foreseeable uses, not just intended uses, Article 5(1)(b) covers both objective and effect.

What good looks like: You've conducted a structured assessment using the European Commission's Guidelines on prohibited artificial intelligence practices, documented your analysis, and can demonstrate that the system neither intends nor produces material behavioral distortion that causes significant harm to vulnerable users.

2. General-purpose system misuse review

If you operate a General-Purpose AI Model, assess whether it's reasonably likely to be used in a manner prohibited by Article 5, even if that's not your intended purpose.

• Identify usage patterns where users seek mental health support.
• Evaluate whether the system's responses could exploit vulnerability (e.g., through excessive agreeability or sycophancy).
• Implement controls to prevent prohibited uses if risks are identified.

What good looks like: You have monitoring in place to detect therapy-related usage patterns, documented evidence that the system doesn't exploit vulnerabilities, and controls (like usage restrictions or response guardrails) if prohibited uses are reasonably likely.

High-Risk System Obligations (If Applicable)

3. Medical Device Regulation classification

Determine whether your AI system qualifies as a medical device under the Medical Device Regulation requiring third-party conformity assessment (Annex I AI Act).

• Review instructions for use, promotional materials, and public statements for claims about diagnosis, prevention, monitoring, prediction, prognosis, treatment, or alleviation of mental illness.
• Distinguish between "well-being purposes" (not a medical device) and managing depression or other mental health conditions (likely a medical device).
• Consult Medical Device Coordination Group guidance on mental health applications.

What good looks like: You have a documented classification decision with clear rationale, supported by review of MDR criteria and consultation with regulatory specialists if the determination isn't straightforward.

4. High-risk compliance (if classified as medical device)

If your system is a medical device requiring third-party conformity assessment, implement the full high-risk AI system requirements.

• Establish a risk management system (ongoing identification, analysis, evaluation, and mitigation).
• Implement data governance practices covering training, validation, and testing data.
• Prepare Technical Documentation (Annex IV) covering system design, development, and performance.
• Enable automatic recording of events (logs) throughout the system lifecycle.
• Design for human oversight appropriate to the risk level.
• Conduct conformity assessment procedure before placing on market.

What good looks like: You have documented evidence for each requirement, a quality management system in place, and completed conformity assessment with a notified body before market placement.

Transparency Obligations

5. Disclosure of AI interaction

Ensure users know they're interacting with an AI system unless it's obvious to someone "reasonably well-informed, observant and circumspect."

• Implement clear disclosure at the start of each conversation or session.
• Account for vulnerable groups (children, elderly, people in emotional distress) who may not find AI interaction obvious.
• Don't assume obviousness, the Commission's Guidelines on Article 50 stress that context matters.

What good looks like: Your system explicitly states it's an AI at conversation start, uses plain language appropriate for vulnerable users, and you've documented why this approach satisfies the "obvious" test for your specific context and user base.

6. Continued interaction disclosure

For ongoing conversations, maintain awareness that the user is interacting with AI.

• Periodic reminders in extended sessions.
• Persistent visual indicators in the interface.
• Clear labeling when switching between AI and human support.

What good looks like: Users cannot reasonably forget they're interacting with AI during extended sessions, and you have evidence (user testing, feedback) confirming the disclosure remains effective over time.

Model-Level Obligations (If You're a Model Provider)

7. Systemic risk assessment (for models with systemic risk)

If you provide a General-Purpose AI Model with systemic risk, identify and mitigate risks to public mental health, fundamental rights, and society.

• Assess whether therapy/support usage creates systemic risks.
• Document mitigation measures for identified risks.
• Establish serious incident reporting procedures for harm to mental or physical health.

What good looks like: You have a documented systemic risk assessment covering mental health impacts, implemented mitigations proportionate to the risks identified, and incident reporting processes ready for the 2 August 2026 enforcement date.

Common Mistakes

Assuming "well-being" exempts you from medical device classification. The line between well-being and managing mental health conditions is narrow. Mood-tracking questionnaires and coping exercises can constitute medical devices if they assess, monitor, or manage depression.

Relying solely on intended purpose to avoid prohibited practice screening. Article 5(1)(b) covers both objective and effect. If your general-purpose system is reasonably likely to exploit vulnerabilities, you're responsible even if that wasn't your intent.

One-time disclosure at signup. Article 50 transparency obligations apply to each interaction. A disclosure buried in terms of service doesn't satisfy the requirement that users be aware during the interaction itself.

Ignoring model-level obligations when you only think about system compliance. If you place an AI system on the EU market that integrates your own model, that model is also considered placed on the market (Recital 97). You face obligations at both layers.

Next Steps

1. Complete this checklist for each AI system that could be used for therapy or emotional support, including general-purpose systems where this use is reasonably foreseeable.
2. Document your classification decisions with clear evidence trails, regulators will expect you to show your work.
3. Establish ongoing monitoring for usage patterns that might trigger prohibited practices or medical device classification.
4. Prepare for 2 August 2026, when the European Commission can begin enforcement actions.
5. Review the Commission's Guidelines on prohibited practices and Article 50 transparency, these bind the Commission's enforcement approach and give you the clearest view of how rules will be applied.

If your assessment identifies a prohibited practice, you cannot place the system on the market or put it into service in the EU. If it's a high-risk medical device, you need conformity assessment before market placement. There's no grace period for these requirements, they're in force now.