[Interspeech 2025] Tai-LALM: Scaling the "Acoustic Long-Tail" with TW-Sound580K

TL;DR

Researchers from National Taiwan University and SJTU have tackled the "localization gap" in audio AI. They released TW-Sound580K, a specialized dataset for Taiwanese dialects and soundmarks, and developed Tai-LALM. By using a novel Verify-Generate-Critique (VGC) pipeline and Dual-ASR Arbitration, they boosted regional audio reasoning accuracy by 6.5%, outperforming general-purpose models like Qwen2.5-Omni.

The Problem: Cultural Deafness in Global Models

Mainstream Large Audio-Language Models (LALMs) are trained on massive, homogenized datasets like LibriSpeech or WenetSpeech. While they excel at standard Mandarin or English, they often perceive regional prosody (like Taiwanese Hokkien) or local environmental sounds as "noise." This leads to Acoustic Hallucinations, where the model forcedly transcribes a local bird call or a specific dialectal emphasis into nonsensical standard text.

The challenge isn't just a lack of data; it's the noise in the labels. Standard ASR systems are brittle when faced with regional variants, making automated dataset creation a "garbage in, garbage out" cycle.

Methodology: High-Fidelity Curation & Perceptual Arbitration

The authors argue that regional adaptation is a data-centric problem. They introduced two core innovations:

1. The VGC Protocol (Verify-Generate-Critique)

To build TW-Sound580K, they didn't just crawl YouTube; they used a sophisticated filter:

• Verify: Two different ASR engines (Whisper and SenseVoice) checked for consistency. If they disagreed too much, the sample was pruned.
• Generate: A powerful teacher model (Gemini-2.5-Pro) generated descriptions based only on the audio, preventing the model from just "reading" bad ASR text.
• Critique: The teacher model reviewed its own output to ensure every description was grounded in actual acoustic evidence.

2. Inference-Time Perceptual Arbitration

Even a well-trained model can be tripped up by a bad ASR transcription during inference. Tai-LALM uses Acoustically-Conditioned Perplexity (AC-PPL). The model looks at multiple ASR transcriptions and asks: "Given the raw audio features (z), which text candidate (h) makes the most sense to my internal brain?" It chooses the one that minimizes uncertainty.

Figure 1: The Tai-LALM pipeline, highlighting the Dual-ASR arbitration and VGC curation.

Experimental Results: SOTA in Regional Reasoning

Evaluated on the TAU Benchmark (specifically designed for Taiwanese cultural sounds), Tai-LALM reached 49.1% accuracy.

• Vs. Zero-shot: +6.5% improvement over the DeSTA 2.5 baseline.
• Vs. Scale: Even when trained on "raw" unfiltered data, the model only reached 46.4%. The VGC curation protocol provided the critical 2.7% boost needed to reach SOTA.
• Generalization: Interestingly, Tai-LALM didn't "forget" how to be a general model. Its performance on LibriSpeech actually improved, suggesting that learning complex regional acoustics helps the model's overall ear for detail.

Table 1: Tai-LALM outperforms larger models like Qwen2.5-Omni (7B) on localized tasks.

Deep Insight: Is Data Purity the New Scaling Law?

The most striking takeaway is that rigorous curation (VGC) was as impactful as the inference logic. In the world of regional dialects where "ground truth" is hard to find, using a multi-model "committee" to verify data is more effective than simply throwing more hours of noisy audio at the GPU.

Limitations & Future Work

The system currently relies on two ASR engines, which adds VRAM overhead. The authors aim to optimize this for edge devices. Furthermore, the goal is to move from "What is this sound?" to "What is the cultural intent behind this dialectal prosody?"—the next frontier in truly culturally-aware AI.

Conclusion

Tai-LALM proves that we don't need 100B parameter models to solve localization. By focusing on the "Acoustic Long-Tail" and implementing smart, modality-aware filtration, we can make AI bridge the gap between global technology and local culture.