Scope - What This Guide Covers
This guide tackles a specific challenge in AI fairness: obtaining demographic data for bias testing when direct collection isn't possible. You'll find practical steps for using data intermediaries and demographic proxies, along with compliance checkpoints related to the Equality Act 2010 and GDPR.
This guide does NOT cover:
- General bias mitigation techniques
- Model Cards or System Cards for fairness documentation
- Fairness metrics selection
- Algorithmic fairness research methods
If you're deploying AI systems in areas covered by the Equality Act 2010, like service provision or employment, and need demographic data to assess potential bias, this guide is for you.
Key Concepts and Definitions
Protected characteristics under the Equality Act 2010 include age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, and sexual orientation.
Demographic data intermediary: An entity that facilitates the sharing of demographic data between data subjects and service providers. It manages consent, access controls, and data governance.
Demographic proxy: Attributes you already have that correlate with protected characteristics. Examples include forename (gender), postcode (race and socioeconomic status), or language preference (national origin).
Bias detection vs. bias mitigation: Detection identifies disparate outcomes. Mitigation changes the model or decision process to reduce disparities. Your data access strategy will differ for each.
Requirements Breakdown
Legal Requirements
GDPR Article 5(1)(b) - Purpose Limitation: Specify "bias monitoring and fairness assessment" as the lawful purpose when collecting demographic data.
GDPR Article 9 - Special Category Data: Most protected characteristics are special category data. You need an Article 9(2) condition (explicit consent, substantial public interest, etc.) and an Article 6 lawful basis.
Equality Act 2010 - Positive Action: Section 158 allows the collection of protected characteristic data to identify disadvantage or underrepresentation, supporting bias detection as a lawful purpose.
Ethical Requirements
Your approach must address:
- Autonomy: Users control whether their demographic data is collected and how it's used.
- Transparency: Clearly disclose what you're testing for and why.
- Representation: Consult with affected communities, especially marginalized groups.
- Privacy: Minimize individual-level data exposure.
Organizational Requirements
- Data Protection Impact Assessment: Required for processing special category data at scale.
- Stakeholder Engagement: Document consultation with user groups affected by potential bias.
- Validation Evidence: Maintain records showing bias testing methodology and results.
- Vendor Due Diligence: Assess data governance practices if using intermediaries or proxy inference tools.
Implementation Guidance
When to Use Data Intermediaries
Consider intermediaries when:
- Demographic data collection isn't standard in your sector (insurance, lending, recruitment).
- Users are skeptical about sharing data directly.
- You lack internal expertise for special category data governance.
- You need data from external populations.
Implementation steps:
- Identify if a demographic data intermediary exists for your sector.
- Assess the intermediary's consent management, access controls, and audit capabilities.
- Define the demographic attributes you need and for which user cohorts.
- Establish contractual data processing terms under GDPR Article 28.
- Implement technical controls to ensure data is used only for bias testing.
When to Use Proxies
Use proxies when:
- Bias can be more accurately identified using a proxy.
- Inferences are drawn at aggregation levels where no individual is identifiable.
- No better alternative exists.
Implementation steps:
- Document why direct collection isn't feasible.
- Select proxies with known correlation to the protected characteristic.
- Validate proxy accuracy for your specific population.
- Aggregate inferences to prevent individual-level identification.
- Conduct a Data Protection Impact Assessment for proxy inference.
- Implement safeguards: human review of results, prohibition on using proxy data as model inputs, time-limited retention.
Proxy accuracy caveat: Ensure proxy performance is validated for your user population.
Common Pitfalls
Pitfall 1: Assuming demographic data collection is prohibited Reality: GDPR allows collection of special category data with a lawful basis and an Article 9(2) condition. Bias monitoring under the Equality Act 2010 can satisfy both.
Pitfall 2: Using proxies to make decisions instead of detect bias If you infer gender from forename for bias testing, that's detection. Using inferred gender in the model itself creates a discrimination risk.
Pitfall 3: Treating intermediaries as a compliance shortcut Intermediaries don't eliminate your GDPR obligations. You're the data controller; the intermediary is your processor.
Pitfall 4: Skipping stakeholder engagement Bias testing without input from affected communities often misses context-specific harms. Consult with relevant organizations before designing tests.
Pitfall 5: Confusing aggregation with anonymization Aggregating proxy inferences doesn't automatically anonymize data under GDPR. If individuals can be re-identified, it's still personal data.
Quick Reference Table
| Scenario | Recommended Approach | Key Compliance Step | Risk to Mitigate |
|---|---|---|---|
| Insurance pricing model, testing for age bias, direct collection not standard | Data intermediary (if available) or direct collection with clear purpose | Article 9(2)(g) substantial public interest + Equality Act 2010 positive action | User skepticism about data misuse |
| Recruitment AI, testing for gender bias, high user trust concerns | Forename-to-gender proxy at aggregated level | DPIA covering proxy inference as special category processing | Proxy inaccuracy for non-Western names |
| Credit decisioning, testing for racial bias, no demographic data held | Postcode-to-ethnicity proxy or intermediary | Article 9(2)(g) + prohibition on using proxy as model input | Re-identification risk if combined with other attributes |
| Public service chatbot, testing for disability bias, accessibility context | Direct collection with transparency and consent | Article 9(2)(a) explicit consent + clear opt-out | Perception of surveillance by disabled users |
| Vendor-provided model, testing for bias, no access to training data | Request Model Card from vendor or conduct proxy-based testing on your deployment data | Vendor Due Diligence + contractual right to audit for bias | Vendor unwillingness to disclose fairness metrics |
Next actions: If you're planning a bias testing project, start by documenting why you need demographic data and whether direct collection is feasible. If it's not, assess intermediary availability in your sector before defaulting to proxies. Every proxy-based approach requires a DPIA and validation evidence showing the proxy's accuracy for your population.
