Predicting Your Pay: Bridging the Information Gap in Crowdsourcing
Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform
The paper introduces a machine learning framework for predicting hourly earnings (HE) and task completion time (TpM) for workers on crowdsourcing platforms. By utilizing a gradient boosting decision tree model (CatBoost) on a dataset of 18 million tasks, the authors achieve SOTA performance in personalized worker-task matching.
TL;DR
Researchers from Yandex and Huawei have tackled a major pain point in the gig economy: workers don't know how much they'll actually earn per hour until they've already wasted time on a task. By using CatBoost to analyze 18 million tasks, they developed a system that predicts hourly earnings with 25% more accuracy than previous methods. When deployed on Yandex.Toloka, it boosted worker retention by 7.4%.
Background: The Marketplace Information Asymmetry
In a typical crowdsourcing marketplace like Amazon Mechanical Turk or Yandex.Toloka, workers are presented with a list of tasks. While the reward per task is visible (e.g., $0.05 per image), the actual effort is hidden. A "simple" task might take 5 minutes, while a "hard" one takes 30 seconds. This asymmetry makes "hourly wage"—the metric workers actually care about—invisible at the point of choice.
The authors argue that the larger a market grows, the harder it is for users to navigate it. Their survey of 5,402 users confirmed this: 68% participate primarily for money, and hourly earnings is a top-tier factor in task selection.
The "Learning Effect" and Heterogeneity
Why is this a hard machine learning problem? Two main reasons:
- Heterogeneity: Not every worker is the same. A worker specializing in audio transcription will be far faster at it than a newcomer, even if the newcomer is generally high-performing.
- The Learning Effect: Users are slow when they first start a new project. As they "master" the interface and instructions, their speed increases significantly.
The figure above demonstrates that for the first 3-5 tasks, completion time is significantly higher, eventually stabilizing as the user gains proficiency.
Methodology: Beyond Simple Averages
The authors moved beyond "Baseline-P" (which just averages past performance) to a feature-rich CatBoost model. Their feature engineering focused on three pillars:
- Project Context: Instruction length, readability scores (FK-index), and technical complexity (number of HTML inputs/buttons).
- Worker History: Dynamic "rolling windows" (last 20/100 tasks) tracking the user's recent accuracy and speed.
- The Visit Context: Is this the first time the user has seen this project? What device are they on?
Why "Cross-Target" Learning Fails
A key insight in the paper is that you cannot simply predict completion time and then divide reward by that time to find the wage. The authors proved that a model specifically trained to optimize for Hourly Earnings (HE) outperforms a "time-derived" model by as much as 128-299% in terms of RMSE. Pay and time are correlated, but the noise in completion time makes it a poor proxy for financial stability.
Experimental Results & Impact
The experiments were conducted on a massive scale: 161,000 users and 767 different projects.
The results (Table 2) show that CatBoost significantly outperforms all baselines. In the most difficult "Cold-Start" scenarios—where a worker encounters a brand-new project—the model reduced error in logarithmic completion time by 32.1%.
Real-World Deployment
When implemented in the Yandex.Toloka production environment, the impact was immediate:
- Retention: New users were 7.4% more likely to return the following week.
- Utility: The "Sort by Hourly Earnings" feature quickly became the second most popular sorting method on the entire platform.
Critical Insight: The Future of Gig Work
This paper serves as a blueprint for platform owners. By using features like instruction readability and HTML complexity, platforms can predict the "difficulty" of a task before a single human touches it.
However, there is a limitation: the "Learning Effect" is modeled as a categorical slice (CS-LEF) rather than a continuous curve. Future work could potentially incorporate Recurrent Neural Networks (RNNs) or LSTMs to better model the rate at which a specific worker learns a specific task type.
Conclusion
By placing the worker's needs (predictable earnings) on par with the requester's needs (fast results), platforms can create a more sustainable ecosystem. This research proves that high-fidelity performance prediction isn't just a technical achievement—it's a massive boost to user loyalty and marketplace efficiency.