CSQuaRE: Bridging Requirements Engineering and Crowdsourcing for Robust Quality Control

Executive Summary

TL;DR: Information Security Q&A platforms like StackExchange often suffer from "low-signal" noise and voting latency. CSQuaRE is a framework that imports the rigor of Software Requirements Engineering—specifically the 3Cs (Completeness, Consistency, Correctness)—to automatically evaluate and score crowd responses using a dynamic Knowledge Base (KB).

In the landscape of crowdsourcing, this work represents a shift from subjective human-in-the-loop voting toward objective, ontology-driven automated assessment.

Problem & Motivation: The Failure of Majority Voting

While crowdsourcing has grown exponentially, its quality control mechanisms remain primitive. Standard platforms rely on "The Wisdom of the Crowd" via majority voting. However, the author finds two critical flaws:

1. Latency: A correct answer might wait days to be "voted up," leaving the seeker empty-handed initially.
2. Accuracy Gaps: A survey of 212 IT professionals revealed that 76% consider crowdsourced answers to be of poor quality. In a sample of 50 phishing-related questions on Quora, over 77% of responses were essentially irrelevant.

The author's Insight is that a crowdsourced response is essentially a "requirement" for information. Therefore, the same metrics used to judge the quality of software requirements can be used to judge the quality of a worker's answer.

Methodology: The 3Cs of CSQuaRE

The core of CSQuaRE is a three-pronged mathematical approach to scoring a response, scaled from 0 to 10.

1. Adequate Completeness ($A{C}_P$)

Instead of checking if an answer is "long," $A{C}_P$ checks for concept coverage. It uses an ontology to see if the response covers the necessary sub-domains. For example, a "complete" answer about Information Security must mention Confidentiality, Integrity, and Availability.

2. Adequate Consistency ($A{C}_N$)

Consistency measures "conflict-free" information. It assesses:

• Textual Cohesion: Ensuring the response isn't just a "bag of words" but a logical sequence of sentences.
• Worker Credibility: Integrating the worker’s past performance and even external reputation (e.g., Twitter data) into the score.

3. Adequate Correctness ($A{C}_R$)

Correctness is not binary (True/False) but a degree of match. By using Word2Vec and FrameNet, the system compares the response against a curated Knowledge Base of 934,000+ security URLs categorized by ISO 27001 standards.

Figure 1: The schematic flow of the CSQuaRE quality control approach.

The Temporal Effect: Re-Calibration

One of the most sophisticated aspects of this paper is the acknowledgment of Temporal Decay. In technical fields like Information Security, an answer that was "Correct" in 2010 (e.g., using SHA1) might be "Incorrect" or "Incomplete" in 2024.

CSQuaRE proposes to re-calibrate historical scores automatically as the Knowledge Base grows, ensuring the platform remains a reliable source of truth over time.

Experiments & Evaluation Plan

The author plans to validate CSQuaRE by:

• Crawing StackExchange data related to ISO 27001 control groups.
• Comparing the automated CSQuaRE Score against scores provided by human Security Experts.
• The hypothesis is that CSQuaRE will show high correlation with expert ratings, effectively automating the "expert review" process.

Critical Analysis & Conclusion

Takeaway

CSQuaRE is a significant step toward making crowdsourcing "sustainable" by providing instant feedback. By leveraging Requirements Engineering, it provides a structured way to handle the inherent ambiguity of human-generated text.

Limitations

• Cold Start for Ontology: The system is only as good as its Knowledge Base. Building a comprehensive ontology for every domain is a massive undertaking.
• External Credibility: Relying on Twitter IDs for credibility introduces privacy and API dependency risks.

Future Outlook

This framework is highly relevant in the age of LLMs. As we move toward "AI Crowdsourcing," where multiple models generate responses, frameworks like CSQuaRE could provide the necessary guardrails to ensure LLM outputs remain consistent and correct over time.