FTI: Bridging the Gap Between Privacy and Precision in Distributed Crowdsourcing

TL;DR

Federated Truth Inference (FTI) is a new framework that allows multiple crowdsourcing platforms to collaboratively find the "ground truth" of regression tasks without ever sharing individual worker answers. By leveraging a distributed Block-EM algorithm, it achieves centralized-level accuracy while keeping data strictly local and reducing communication costs.

Context: Why Centralized Crowdsourcing is Failing

In the era of data privacy (GDPR, HIPAA), the old model of "send all data to the cloud" is increasingly untenable. Imagine three different hospitals crowdsourcing the labeling of X-ray images. They cannot share raw labels due to patient confidentiality, yet they need to combine their workers' insights to filter out noise and unreliable non-experts.

Existing truth inference algorithms (like Majority Voting or standard EM) assume a "God's eye view" of all data. In a distributed world, we face a dual challenge: Privacy (don't share labels) and Efficiency (don't clog the network).

Methodology: The Federated EM Approach

The authors model worker responses using a Gaussian Mixture Model (GMM). Every worker's response is treated as the ground truth plus some noise, where the noise level depends on the worker's latent reliability level.

1. The FTI Algorithm

Instead of sending raw responses $x_{n,m}$, each platform sends aggregated statistics.

• Local Stage: Platforms calculate how likely a worker belongs to a certain reliability tier based on current global estimates.
• Global Stage: The coordinator sums up these local aggregates to update the global truth ${\hat{a}}_m$ and the variance ${\sigma }_l^2$.

Fig 1: The FTI architecture featuring local platforms and a central coordinator.

2. Communication Efficiency (FTI-LE)

Sending updates every single iteration is expensive. The FTI-LE (Local Estimates) variant allows local platforms to run $T$ iterations of EM internally using their local data before syncing with the center. This "lazy" sync mirrors the approach found in modern Federated Learning (like FedAvg), significantly lowering the number of messages required for convergence.

Experimental Validation

The authors tested FTI against centralized EM (C-EM) and specialized models like TBP (Truth, Bias, and Precision).

• Accuracy: In synthetic tests, FTI performed identically to centralized EM, proving that the distributed decomposition results in zero loss of information for the inference task.
• Robustness: Even when only a fraction of workers were assigned to each task, FTI maintained a low RMSE, whereas simple Averaged Local Estimates (ALE) failed completely due to data sparsity.

Fig 2: RMSE performance vs. the density of workers per task. Note how FTI stays robust compared to ALE.

Critical Insight & Conclusion

The brilliance of FTI lies in its use of Sufficient Statistics. The EM algorithm for GMMs is naturally "summable." By recognizing that the global M-step only requires sums of local E-step outputs, the authors created a privacy-preserving protocol that is mathematically equivalent to the centralized version.

Takeaway: FTI represents a significant step for decentralized intelligence. It proves that you don't need to "see" everyone's data to know who is telling the truth.

Limitations: The convergence of FTI-LE (with multiple local steps) is not theoretically guaranteed, though it works well in practice. Future work might explore asynchronous updates where platforms don't have to wait for each other to finish their local rounds.