Beyond Micro-Tasks: Orchestrating the Future of Crowdsourcing Processes
9554_Crowdsourcing Processes- A Survey of Approaches and Opportunities.
This article surveys the transition from simple micro-tasking to complex "crowdsourcing processes," proposing a multi-dimensional framework to evaluate tools capable of coordinating intricate workflows involving both human workers and machine tasks. It reviews 11 state-of-the-art platforms, including TurKit, AutoMan, and CrowdComputer, highlighting their ability to manage structured work beyond basic outsourcing.
TL;DR
The crowdsourcing landscape is shifting from simple, independent micro-tasks to complex, multi-stage Crowdsourcing Processes. This article taxonomizes the design space for these processes—covering definition paradigms, data flow, and quality control—and evaluates 11 pioneering platforms that attempt to automate the coordination between global crowds and machine algorithms.
The "Structured Work" Bottleneck
In the early days of human computation, the focus was on the "micro-task": labeling one image or translating one sentence. Platforms like Amazon Mechanical Turk thrived on this simplicity. However, real-world problems—like writing a research article or planning a cross-country trip—cannot be solved by a single worker in a vacuum.
The authors argue that such tasks require structured work: a sequence of interdependent steps where the output of one worker (e.g., writing a draft) becomes the input for another (e.g., editing) or a machine (e.g., automated grammar checking). Without specialized process management tools, researchers are forced to manually manage data transfers, monitor worker quality, and handle exceptions, leading to significant overhead.
Methodology: The Five Dimensions of Crowdsourcing Logic
To evaluate how tools handle this complexity, the authors propose a robust framework consisting of five analysis dimensions:
- Process Definition: Is the workflow defined via code (Imperative like TurKit), SQL queries (Declarative like CrowdDB), or a GUI (Visual like CrowdComputer)?
- Task Support: How does the system distinguish between crowd tasks (human effort) and machine tasks (automated scripts/web services)?
- Control Flow: Does the tool support parallel execution, loops/iterations (essential for multi-pass editing), and decision gateways?
- Data Management: How is data passed? "By value" (copying data) or "By reference" (sharing a pointer)?
- Quality Control: What built-in patterns exist? Examples include Consensus (multi-worker agreement), Voting, and Gold Standards (control questions).
Figure 1: Conceptual overview of the dimensions required to manage structured crowdsourcing processes.
A Competitive Landscape: From TurKit to CrowdDB
The paper provides a comprehensive comparison of 11 platforms. A few standouts illustrate the diversity of the field:
- TurKit: A JavaScript-like environment that allows developers to treat humans as "functions" in a program.
- AutoMan: A Scala-based framework that treats the crowd as a typed variable, automatically managing the budget and confidence levels.
- CrowdComputer: A visual tool based on BPMN (Business Process Model and Notation), making it accessible to business analysts rather than just software engineers.
Table 1: Comparison of state-of-the-art platforms across definition paradigms and features.
Critical Analysis: Why Aren't These Tools Ubiquitous?
While the research is promising, the authors identify several "weak links" that prevent these tools from reaching mainstream industrial adoption:
- Integration Complexity: Many tools use proprietary languages (like "Dog" in Jabberwocky) that are difficult to integrate with existing enterprise IT stacks.
- Narrow Quality Control: Most systems still only control quality at the task level. True process management requires Adaptive Crowdsourcing—where the entire workflow changes its route dynamically based on the quality of intermediate results and remaining budget.
- Human Factors: These tools often ignore worker training and motivation. A "machine-centric" view of human workers fails to account for learning effects or worker fatigue.
Conclusion and Future Outlook
The shift from "crowdsourcing as outsourcing" to "crowdsourcing as a process" is inevitable. For practitioners, the takeaway is clear: stop thinking about individual tasks and start designing pipelines. The future of the field lies in Adaptive Processes—systems that can intelligently decide when to involve an expert, when to loop a task for better quality, and when to let a machine take over, all while maximizing efficiency.
As a field, we are moving toward a "Community Effort" to standardize these frameworks, with the authors even proposing a Wikipedia-based evolution of this taxonomy to keep pace with the rapid development of human-AI collaboration.