NVBench: Decoding the "Human" in AI Speech through Non-Verbal Vocalizations

TL;DR

While modern TTS systems have mastered what we say, they often struggle with how we say it—specifically regarding the non-lexical sounds like gasps, chuckles, and sniffs that define human emotion. NVBench is a new bilingual benchmark that introduces a 45-type taxonomy and a rigorous "multi-axis" evaluation protocol to solve the lack of standardized testing for these subtle but vital paralinguistic cues.

The Motivation: Why Speech Quality Isn't Enough

For years, the gold standard for TTS was Naturalness and Intelligibility. However, as we move toward "super-human" speech generation, there is a glaring "Affective Gap." Current systems might sound crystal clear (high DNSMOS) but fail to sob naturally when a prompt demands distress, or they might ignore a "[sigh]" tag entirely.

The authors identified that existing benchmarks are often highly skewed toward "easy" sounds like laughter, ignoring low-energy oral cues (tongue clicks, lip smacks) or long-duration states (trembling speech). NVBench addresses this by disentangling Speech Quality from NVV Controllability.

Methodology: The 45-Cues Universe

NVBench is built upon a structured taxonomy that mirrors human physiology and social interaction. By dividing vocalizations into six categories—Respiratory, Throat, Laughter, Crying, Emotional, and Oral—the benchmark forces models to be judged on their breadth of expression.

Data Construction Pipeline

The researchers didn't just synthesize data; they used a three-stage "human-in-the-loop" process:

1. Seed Mining: Identifying real-world acoustic patterns from expressive human datasets (InstructTTSEval).
2. LLM-Driven Generation: Using Gemini 2.5 Pro to generate 4,500 bilingual instances across the 45 types.
3. Rigid Validation: Ensuring each sample was contextually plausible and the NVVs were perceptually salient.

The Multi-Axis Evaluation

The heart of NVBench is its evaluation protocol. Instead of a single "Mean Opinion Score" (MOS), it measures:

• Intelligibility: WER/CER (to ensure NVVs don't break the transcript).
• Controllability: Precision/Recall of the intended NVV tags and Normalized Tag Distance (NTD) to check if the sigh happened at the right time.
• Perceptual Salience (NVV PE): Assessing if the vocalization sounds authentic or like a robotic artifact.

Key Insights from 15 SOTA Systems

The study benchmarked both tag-based (e.g., ElevenLabs, ChatTTS) and prompt-based (e.g., GPT-4o mini, Gemini) systems.

The Findings:

1. The Intelligibility-Quality Split: Prompt-based models like Gemini often produce incredibly natural-sounding laughs but suffer from "prompt leakage"—repeating text or hallucinating extra "ha-ha-ha" segments that spike the Word Error Rate (WER).
2. Selective Compliance: Many systems (like ChatTTS) excel at a few specific sounds (laughter) but have a "coverage gap" for others.
3. Persistent Bottlenecks:
   • Low-SNR cues: Lip smacks and swallows are often masked by noise or ignored.
   • Long-horizon Affect: Sustained sobbing or crying requires a temporal coherence that most current models lack.

Figure: The "Coverage Gap" is visible in the sparse heatmaps of tag-based systems compared to the dense but varying performance of prompt-based models.

Experimental Performance

As shown in Table 3 of the paper, ElevenLabs emerged as a leader in balancing coverage with audio quality, while Orpheus TTS showed impressive precision in tag placement. Interestingly, the study found that adding NVVs actually decreases scores on standard objective metrics like DNSMOS, highlighting the urgent need for "NVV-aware" evaluators.

Critical Analysis & Conclusion

NVBench proves that the "uncanny valley" of AI speech is no longer about the signal quality, but about the paralinguistic timing and texture.

Limitations: The reliance on LLMs (Gemini) as "Multi-Raters" is efficient but may carry its own biases toward "smoothness" rather than "rawness." Furthermore, the benchmark is currently limited to English and Chinese.

Future Outlook: For TTS to reach true emotional intelligence, researchers must move beyond "tag-insertion" and focus on trajectory control—modeling how a sob affects the timbre of every subsequent word in a sentence. NVBench provides the map; now the community must build the engine.