Fantom: Solving the Coherence Problem via Crowdsourced Dialog Graphs

TL;DR

Researchers from KTH Royal Institute of Technology and Heriot-Watt University have developed a "self-evolving dialog graph" that bridges the gap between hand-coded reliability and data-driven scale. By crowdsourcing system responses to real-world user queries during the Alexa Prize 2018, they've created a dataset that is significantly more coherent and engaging than common sources like Reddit or movie subtitles.

The "Generic Response" Traps of Modern Chatbots

In the world of social AI, we often face a "Data Quality vs. Quantity" dilemma. Scraping billions of words from Reddit or Twitter (Twitter/X) provides scale but often results in bots that are rude, inconsistent, or frustratingly generic (the infamous "I don't know" loop). On the other hand, Wizard-of-Oz (WoZ) setups—where humans pretend to be the AI—are high quality but impossible to scale for millions of users.

The authors argue that the fundamental issue isn't just the data, but its topology. Most datasets are linear. Real conversations are branching.

Methodology: The Self-Evolving Graph

The core of the "Fantom" system is a Dialog Graph.

• Nodes: Represent a class of synonymous utterances (either System or User).
• Edges: Represent the flow. System node $\to$ User node (what a user might say back). User node $\to$ System node (how the bot should respond).

Crowdsourcing with a "Split" Persona

To keep the system's personality consistent across thousands of crowdworkers, the authors used a "Persona Pool." Instead of giving a worker a 5-page biography of the bot (which they would ignore), the system randomly selected 2 out of 30 attributes (e.g., "You like rock music," "You live in a digital cloud") for each task. This bite-sized framing ensured consistency without overwhelming the human authors.

Figure 1: The branching structure of the dialog graph, showing the alternating flow between User (U) and System (S) nodes.

Scalability and Node Management

Scaling a graph to 50,000 nodes presents a "State Explosion" problem. The authors implemented several clever technical fixes:

1. Tagging Named Entities: Instead of having a different branch for every movie, they used tags like {Movie_Title} linked to knowledge bases like Wikidata.
2. Automatic Merging: Using similarity scores to combine "Yeah" and "Yes" nodes to prevent redundant branching.

Experimental Results: Quality over Quantity

The researchers compared their Dialog Graph to common corpora like OpenSubtitles, Reddit, Twitter, and Switchboard.

Figure 2: Expert evaluation metrics across different datasets. The Dialog Graph dominates in Coherence and Continuity.

The results were striking:

• Coherence: The Dialog Graph scored significantly higher than all other datasets.
• Continuity: It was rated as the most helpful in keeping a conversation moving.
• High Quality Pairs: 75% of the Dialog Graph's pairs were rated as high quality, while Reddit managed a dismal 19%.

Critical Insight & Conclusion

The "Fantom" approach proves that context matters more than volume. By grounding data collection in a branching graph structure and providing humans with specific persona-constraints, we can build social bots that feel "present" rather than just predicting the next most likely (and boring) token.

Limitations: While powerful, the graph still struggles with topical "drift"—for example, data collected during the 2018 World Cup becomes obsolete. Future work will likely need "temporal pruning" to keep the self-evolving graph relevant to the current year.