Collusion-Proof Crowdsourcing: Distinguishing Expertise from Fraud

TL;DR

Crowdsourcing platforms are increasingly plagued by colluding workers who use duplicated submissions and group plagiarism to earn rewards with zero effort. This paper challenges the traditional assumption of worker independence by introducing a "Collusion-Proof" mechanism. By analyzing the Worker Performance Change Rate, the system can distinguish between honest experts who agree on a correct answer and malicious groups who agree on a lie, improving result accuracy by over 20% in high-collusion environments.

Background: The Hidden Collaborative Networks

The "Wisdom of the Crowd" relies on diversity. However, workers are not isolated islands; they communicate via social media or operate multiple "Sybil" accounts. This leads to three toxic behaviors:

1. Duplicated Submission: Groups submitting the same answer to guarantee a payout.
2. Group Plagiarism: Low-effort workers copying a single person's work (or a coin flip).
3. Sybil Attacks: One person masquerading as many to dominate the vote.

Traditional aggregation methods like Majority Voting (MV) are easily "poisoned" by these behaviors because they cannot tell if 5 identical answers come from 5 brilliant minds or 1 person with 5 accounts.

The Problem & Motivation: When is Agreement Suspect?

Prior work focused on "Spatial Crowdsourcing" (using GPS proximity to find colluders) or "Rating Tasks" (subjective scores). Neither works for general, objective tasks like image labeling.

The authors' core insight: Expert consensus is robust, while collusion is fragile. If you remove the answers of a group of experts, the remaining crowd's inferred result might weaken slightly, but the logic stays the same. If you remove a group of colluders who forced a wrong answer, the collective accuracy should spike.

Methodology: The Power of the "Change Rate"

The paper proposes a workflow that moves from task scheduling to a specialized detection step before final aggregation.

The core mechanism is the Worker Performance Change Rate. The algorithm:

1. Clusters repeated answers into groups.
2. Calculates the crowd performance (using algorithms like DS or GLAD) with the group included.
3. Re-calculates performance with that group excluded.
4. If the difference exceeds a specific Threshold, the group is purged as a collusion gang.

Modified algorithms like C-MV, C-DS, and C-GLAD were created by wrapping this detection around baseline models.

Experiments & SOTA Comparison

The authors tested their approach against "Findcolluders" (a similarity-based baseline) using synthetic data and real-world datasets (ducks and adult2).

Accuracy vs. Worker Ability

The C-DS model showed a massive performance lead. As worker expertise increases, all models improve, but C-DS remains consistently higher by filtering out noise that confuses standard EM-based models.

Resilience to Collusion Density

A critical test was increasing the proportion of colluders in the crowd. While standard DS accuracy plummets as colluders overwhelm the signal, the Collusion-Proof (C-DS) variant maintains high accuracy even when colluders represent a significant portion of the workforce.

Critical Analysis & Future Outlook

Takeaway: Agreement does not equal truth. This paper provides a mathematically grounded way to treat "consensus" as a variable that must be validated against its impact on the system's global health.

Limitations:

• The Threshold Problem: The current model uses a set threshold (e.g., 0.028). In a truly dynamic environment, this threshold may need to be learned via reinforcement learning or meta-learning.
• Sophisticated Collusion: If colluders intentionally inject some "honest-looking" noise into their answers to avoid being a perfect duplicate, the change rate detection might become less sensitive.

Conclusion: This is a foundational step toward "Adversarial Crowdsourcing," where the system actively protects itself against the increasingly social and coordinated nature of the modern digital workforce.