StreamingTalker: Enabling Real-Time Digital Humans with Autoregressive Diffusion

TL;DR

StreamingTalker introduces an innovative Autoregressive (AR) Diffusion Model for 3D facial animation. By breaking the "process-whole-sequence" paradigm of existing diffusion models, it enables streaming output with a 25ms fixed latency and handles audio of arbitrary length without the usual "drifting" or quality loss seen in traditional models.

The "Beyond Training Horizon" Problem

Most modern 3D facial animation models (like FaceDiffuser or DiffSpeaker) are trained on fixed-length windows (e.g., 4-5 seconds). When presented with a 1-minute audio clip, these models often fail because they lack the mechanism to handle temporal dependencies beyond their training window, or they require the entire audio to be processed before the first frame is rendered. This creates a "latency wall" that blocks any real-time application.

Methodology: The AR Condition Predictor

The core innovation lies in treating diffusion not as a global sequence operation, but as a step-wise conditioned process.

1. VQ-VAE Latent Space: To ensure the diffusion model operates efficiently, raw facial meshes (thousands of vertices) are compressed into a compact latent space.
2. Condition Predictor: A Transformer decoder takes the past h frames of motion and the current audio embeddings (from HuBERT) to predict a "dynamic condition."
3. Lightweight Diffusion Head: Unlike heavy Unets, StreamingTalker uses a single-layer MLP as the diffusion head. Because the condition predictor provides such a strong "prior" based on recent history, a lightweight head is sufficient to recover the details from noise.

The use of ALiBi (Attention with Linear Biases) in the self-attention layer is a strategic choice. It allows the model to "extrapolate" to sequences much longer than those seen in training by biasing the attention toward local, recent history, which is more relevant for speech-driven motion.

Experiments: Superiority in the Long Run

While StreamingTalker performs slightly better on standard short benchmarks, its true strength shines in long-sequence generation.

Quantitative Edge

On the BIWI dataset, for sequences up to 2000 frames:

• StreamingTalker (Ours): LVE 4.4596 | FDD 3.8912
• DiffSpeaker: LVE 5.2213 | FDD 4.7980

The lower FDD (Face Dynamics Distance) indicates that the "upper face" (not just the lips) remains expressive and consistent over time, whereas other models tend to "freeze" or become jittery after the training horizon is exceeded.

Qualitative Precision

Visually, the model captures fine-grained articulatory details. It handles "bilabial consonants" (like /m/, /p/, /b/) by ensuring the lips fully close—a common failure point in previous Transformer-only models that tend to "over-smooth."

Efficiency: The 25ms Guarantee

For any interactive AI (like a digital assistant or virtual NPC), total latency must stay under roughly 100ms. Previous diffusion models took seconds to process long audio. StreamingTalker’s latency is independent of audio length, staying at a flat 25ms. This allows for a smooth 40 FPS rendering loop even on mid-range hardware.

Insights & Future Outlook

StreamingTalker successfully bridges the gap between the diversity of diffusion models and the efficiency of autoregressive models.

Limitations: The model still relies on identity embeddings, meaning it requires fine-tuning or a significant subject pool to generalize to completely unseen speakers' "styles." Additionally, it focuses heavily on lip-sync; adding "emotional conditioning" (e.g., integrating EMOCA) would be the next logical step to move from a "talking head" to a "feeling head."

Takeaway: This work provides a blueprint for deploying high-fidelity generative 3D assets in live-streaming and interactive LLM-integrated avatars.