SS-Crowd: Turning the "Wisdom of Crowds" into Self-Supervision for AI
Collective intelligence as a source for machine learning self-supervision
This paper introduces SS-Crowd, a self-supervision framework that leverages collective intelligence from web Question-Answering (QA) forums like Yahoo! Answers to validate machine learning beliefs. Specifically, it integrates with the Never-Ending Language Learner (NELL) to verify first-order logical rules induced by its Rule Learning (RL) algorithm through human common sense.
TL;DR
Researchers have developed SS-Crowd, a system that allows the Never-Ending Language Learner (NELL) to "ask" the internet for help. By converting its internal logical rules into natural language questions on Yahoo! Answers, the system uses the collective intelligence of human users to validate its beliefs. This "Reversed Macro-QA" approach helps bridge the gap between machine-extracted facts and human common sense.
The "Liar's Paradox" in Never-Ending Learning
Systems designed for continuous learning, such as Carnegie Mellon’s NELL, face a fundamental challenge: Cumulative Error. Because NELL learns by reading the web 24/7, if it extracts one slightly wrong fact, it might use that fact to justify thousands of subsequent incorrect beliefs.
Current solutions rely on "Knowledge Integrators" or rare human specialist intervention. However, specialists are expensive, and automated integrators lack the nuanced "common sense" that humans take for granted. The authors identify a gap: why not use the existing, massive human activity on social web forums to provide this "sanity check"?
Methodology: The Reversed Macro-QA Approach
The core innovation lies in flipping the traditional Question-Answering (QA) paradigm on its head.
1. From Logic to Language
NELL’s Rule Learner (RL) generates rules in a Prolog-like syntax. For example:
athletePlaysInLeague(x, NFL) :- athletePlaysForTeam(x, Giants)
SS-Crowd automatically translates these into human-friendly queries: "Is this statement always true? If state X has capital Z and Z is in country Y, then X is in country Y."
2. RHCI: Designing for Machine Consumption
The paper introduces Reverse Human-Computer Interaction (RHCI). While traditional HCI focuses on making technology easy for humans, RHCI focuses on making human output easy for machines.
- The Constraint: Humans often provide complex, conversational answers.
- The Fix: By adding the prefix "Please just answer yes or no", the authors simplified the "signal" for the machine without significantly biasing the human's "common sense" judgment.
Figure 1: The progression of improvement in resolving user opinions, comparing regular vs. Yes/No questions.
3. The Consensus Algorithm
The system doesn't just trust one user. It uses a Macro-QA definition: looking at a set of answers for a single target idea and calculating a score based on:
- Total approvals vs. rejections.
- The "Best Answer" chosen by the community (weighted more heavily).
- Redundancy of simple phrases.
Experimental Insights
The authors tested 60 high-impact rules from NELL. The results revealed a fascinating disconnect between "Web Logic" and "Common Sense":
- The Specificity Trap: Humans were often more restrictive than the system. For a rule stating that NHL teams play hockey, users pointed out that "NHL" could technically stand for "New Hampshire Lacrosse," rejecting the rule.
- The Popularity Bias: A rule like
athletePlaysSport(x, basketball) => athletePlaysInLeague(x, NBA)was rejected by users. While NELL’s limited web-text experience might suggest most famous basketball players are in the NBA, human "collective intelligence" knows there are countless leagues worldwide.
Table 1: Performance metrics showcasing that Yes/No Individual and Combined approaches provide superior F-Measures and Accuracy.
Critical Analysis & Conclusion
Takeaway
SS-Crowd effectively demonstrates that Collective Intelligence is an underutilized resource for self-supervision. By treating the crowd as a "Macro-Reading" source, AI can fix its own internal logic.
Limitations
- Latency: While 90% of answers arrived within 3 hours—impressive for humans—this is still "glacial" compared to typical machine learning inference times.
- Community Dependency: The "points policy" of forums like Yahoo! Answers (where you must give points to ask) remains a bottleneck for full automation.
Future Outlook
As we move into an era of massive LLMs, the concept of RHCI becomes even more critical. Teaching AI not just to answer our questions, but to ask us the right questions to verify its own internal world model, could be the key to more reliable and trustable artificial intelligence.