These questions come from privacy officers, data scientists, and model risk managers who are tasked with making federated learning projects private. They're not asking theoretical questions. They're asking what happens when you try to implement Differential Privacy in a production federated system, and whether the accuracy hit will kill the business case.
Q1: If we add noise for Differential Privacy, how much accuracy are we giving up?
The impact on accuracy depends on your model type and data availability.
For simpler models like linear regression, logistic regression, and decision trees, the privacy-utility tradeoff is manageable. You can add calibrated noise to model updates and often maintain strong performance while meeting privacy guarantees. Teams in the UK-US PETs Prize Challenges demonstrated this with horizontally partitioned data, achieving both high accuracy and formal privacy.
For neural networks and deep learning models, it's more challenging. Larger models require more noise for the same privacy level, and that noise compounds across training iterations. The sheer parameter count works against you.
A practical workaround is to pre-train on public data without Differential Privacy, then fine-tune with Differential Privacy on sensitive data. Research by Li et al. showed that pre-trained language models fine-tuned with Differential Privacy can achieve nearly the same accuracy as models trained without any privacy protection. This is important for language models, image recognition, and other domains where you can use publicly available datasets.
The catch: you're not protecting the public pre-training data. Ensure your legal team has cleared the use of that data and that it doesn't contain personal information you're obligated to protect.
Q2: Who actually adds the noise in a federated setup?
In centralized training, the server adds noise during training. In federated learning, it's more complex because you don't want to trust a single party with raw model updates.
For horizontally partitioned data (where participants hold different rows of the same table), each participant adds noise to their local model update before sending it to the aggregator. This modification of the FedAvg approach ensures that the aggregated global model has sufficient total noise to satisfy Differential Privacy. This works even if the aggregator is malicious, as they never see unprotected updates.
For vertically partitioned data (where participants hold different columns about the same individuals), it's messier. You can't add noise before entity alignment, as it will prevent attributes from matching correctly across participants. The noise must be added after alignment, which means either trusting one participant to add it or using Homomorphic Encryption or Secure Multi-Party Computation to add noise collectively without any single party seeing the unprotected result.
Q3: How do we know if we've added enough noise?
You need to set a privacy budget, typically expressed as epsilon (ε) in Differential Privacy literature. Lower epsilon means stronger privacy but more noise. Your choice depends on your risk tolerance, regulatory requirements, and the sensitivity of the training data.
GDPR doesn't specify an epsilon value, but if you're conducting a Data Protection Impact Assessment for high-risk processing, you'll need to justify your privacy parameters. Document why your chosen epsilon provides adequate protection given the data sensitivity and potential harms.
Track your privacy budget across all model training runs. Differential Privacy guarantees degrade with repeated queries or model updates on the same dataset. If you retrain monthly, your cumulative privacy loss grows unless you add fresh data or increase noise levels.
Q4: What if our accuracy drops below acceptable thresholds?
First, check if you're using the right baseline. Don't compare a differentially private federated model to a centralized model trained on pooled data. Compare it to models trained on each participant's local data separately, or to no model at all.
If accuracy is still insufficient, consider:
- Pre-training on public data to reduce the learning burden on your private fine-tuning phase.
- Increasing your privacy budget (higher epsilon) if your risk assessment supports it.
- Collecting more training data across participants, since larger datasets tolerate more noise for the same privacy level.
- Simplifying your model architecture if you're using deep learning. Sometimes a smaller model with Differential Privacy outperforms a large model that overfits despite the noise.
You might also find that your use case doesn't require federated learning. If participants can share aggregated statistics or synthetic data instead of training a shared model, you may achieve your goals with simpler privacy-preserving techniques.
Q5: Does Differential Privacy protect us during entity alignment in vertical federated learning?
No. Entity alignment, which matches records across participants, happens before you add noise for Differential Privacy. During alignment, participants learn which of their records correspond to records held by other participants. That's a separate privacy risk.
You need input privacy techniques to protect the alignment phase. Secure Multi-Party Computation or Private Set Intersection can perform alignment without revealing unmatched records.
Differential Privacy protects the trained model from leaking training data. It doesn't protect the intermediate steps of federated learning. You need both input privacy and output privacy for a complete solution.
Q6: Can a malicious participant defeat Differential Privacy by not adding noise?
In horizontally partitioned federated learning, a single participant who skips adding noise won't break the system's privacy guarantee, assuming everyone else adds noise correctly. The aggregated model will still contain noise from honest participants.
But if multiple participants collude or if the aggregator is malicious and colludes with participants, they could potentially extract information. Your threat model matters. If you're worried about malicious participants, you need cryptographic verification that noise was added, not just trust.
For high-stakes applications, consider using Secure Multi-Party Computation to verify that each participant's update includes the required noise without revealing the update itself.
Where to go for more
NIST's blog series on privacy-preserving federated learning covers input privacy techniques for both horizontal and vertical partitioning. Their series on Differential Privacy provides deeper technical detail on the privacy framework itself.
The UK-US PETs Prize Challenges published winning solutions that demonstrate practical implementations. Review those if you're building a business case or architecture proposal.
If you're conducting a Data Protection Impact Assessment under GDPR Article 35, document your Differential Privacy parameters, privacy budget tracking, and the technical measures you're using for both input and output privacy. Your DPO will need this for regulatory justification.
