Subjective Quality Assessment in the Wild: A Survey of Crowdsourcing Frameworks

TL;DR

Subjective quality assessment (QoE) is shifting from the laboratory to the web. This paper surveys the ecosystem of web-based frameworks—such as QualityCrowd, CrowdMOS, and In-momento—that allow researchers to conduct rigorous multimedia experiments using global worker pools like Amazon Mechanical Turk. The focus is on how these tools solve the triple challenge of media delivery, methodological rigor, and worker reliability.

Problem & Motivation: The Lab vs. The Crowd

Historically, assessing the quality of a video or audio clip required a controlled lab environment (ITU-R BT.500 standards). While accurate, this is a bottleneck for modern data needs. Crowdsourcing offers scale and "real-world" diversity, but it is a "Wild West" for data quality:

• Hidden Variables: We don't know if a user is watching a 4K video on a 720p screen or listening to high-fidelity audio through cheap laptop speakers.
• Malicious Behavior: "Random clickers" or bots try to finish tasks as quickly as possible for monetary gain.
• Technical Barriers: Synchronizing playback and ensuring uncompressed media reaches the user without network-induced artifacts is technically demanding.

Methodology Detail: The Framework Ecosystem

The researchers categorize the tools into several key players, each with a unique "DNA" for tackling reliability:

1. The Reliability Gatekeepers

• Quadrant of Euphoria: Uses the Transitivity Satisfaction Rate (TSR). If a user prefers A over B and B over C, but then chooses C over A, they are flagged for inconsistency.
• CrowdMOS: Employs the Sample Correlation Coefficient. It compares an individual's rating to the global average; if the correlation is too low, the user is discarded.

2. The Interactive Architects

• QualityCrowd & WESP: These are "Power User" tools. They provide scripting languages and management layers to build complex, multi-stage experiments including training sessions and "gold standard" control questions.

3. The Future: In-momento Testing

Perhaps the most innovative approach discussed is In-momento Crowdsourcing. Unlike older methods that filter data after the experiment (wasting money and time), this framework builds a Live Reliability Profile. If a worker switches browser tabs or skips video segments, they are penalized in real-time and may be barred from continuing.

Fig 1: The hierarchy from worker crowds to specialized QoE frameworks.

Comparing the Frameworks

The strength of this paper lies in its comparison matrix, which serves as a buyer's guide for researchers:

Table 1: Detailed feature comparison (extracted from paper).

Critical Analysis & Future Outlook

While the survey is comprehensive, it highlights a persistent gap: The Hardware Blindspot. Even the best framework can't fully compensate for a worker's poor physical hardware.

Takeaway for the Industry: The industry is moving toward "Crowdsourcing 2.0". We are seeing a transition from "batch processing" (run a test, clean the data) to "dynamic testing" (monitor the user, adjust the task). For product teams developing codecs or streaming algorithms, integrating these frameworks into the CI/CD pipeline could allow for massive-scale, automated human-in-the-loop testing.

Conclusion

There is no "one-size-fits-all" framework. If you need standard ACR video tests, QualityCrowd is the robust choice; if you are worried about budget and worker quality, the In-momento approach is the superior paradigm for the modern web.