Crowdsourcing the Curriculum: Leveraging Human Intelligence for MOOC Recommendations

TL;DR

With the explosion of Massive Open Online Courses (MOOCs), "quality" has become the ultimate filter. However, quality is inherently subjective and difficult for AI to quantify. This paper proposes a crowdsourcing-based approach that uses pairwise student comparisons to rank courses. By introducing the Two-Step Away (TSA) heuristic, the author provides a way to find the best courses that is more accurate than local voting and more efficient than complex mathematical optimization.

The "Subjectivity" Bottleneck

In the world of MOOCs, a computer can easily track the length of a video or the number of participants, but it cannot easily judge a lecturer’s charisma or the clarity of an explanation. These are "soft" attributes that humans perceive instantly but machines miss.

The most reliable way to measure this is through Pairwise Comparisons (e.g., "Is Course A better than Course B?"). However, collecting every possible pair is impossible, and human voters are often "noisy" (they make mistakes or have biased views). The challenge is: How do we reconstruct a perfect quality ranking from a sparse, messy matrix of human votes?

Methodology: Beyond Simple Voting

The paper formalizes this as the Judgement Problem. While the Maximum Likelihood (ML) approach (based on Kemeny rankings) yields the most accurate results, it is a known NP-hard problem—calculating it for a large library of courses would take until the end of time.

The Heuristic Approaches

To solve this, the author shifts focus to graph-based heuristics:

1. Degree Score (DEG): A local approach where a course's rank is based solely on its direct wins and losses.
2. Two-Step Away (TSA): The core innovation. It recognizes that not all "wins" are equal. Beating a high-quality course should count for more than beating a poor-quality one. TSA looks at the "wins and losses" of the objects your target object has interacted with, effectively propagating quality information through the network.

Figure 1: Representation of a vote matrix and its conversion into a directed weighted graph for quality inference.

Experimental Insights

The research tested these algorithms against a "True Ranking" (ground truth) using synthetic data with varying levels of worker accuracy ($p$).

• The Scalability Wall: ML is only feasible for a very small number of objects (e.g., $n=5$).
• The Power of Redundancy: As the "Edge Coverage" (the number of votes per pair) increases, the TSA method's accuracy climbs sharply.
• Performance: In tests with 100 courses, TSA's ability to leverage indirect evidence allowed it to successfully prune out bottom-tier courses and identify the top-k with high precision.

Figure 2: Performance comparison (P@1 and MRR) showing TSA's convergence towards optimal accuracy as more crowdsourced data is collected.

Critical Analysis & Future Outlook

The beauty of the TSA method lies in its independence from the specific error model. Unlike ML, which requires knowing the average worker accuracy ($p$) beforehand—information rarely available in the real world—the heuristics work based on the structure of the data itself.

Limitations: The current model assumes worker errors are independent. In reality, "malicious" workers or "herd mentality" in MOOC forums could introduce correlated errors that might skew a graph-based heuristic.

Takeaway: As we move toward "Human-in-the-loop" AI, algorithms like TSA show that we don't need perfect data or infinite compute. By smartly aggregating "two steps" of human intuition, we can build recommendation engines that truly understand the human definition of quality.