How Can Talking Head Generation Imitate a Person's Speaking Habits in Real Time?

Speaking habit lives in motion space, not pixels

The paper's core claim is that individual speaking habits show up as subtle but consistent lip-shape variations even when two people pronounce the same word. The authors point to differences such as how far the mouth opens on "bus", how wide the lips stretch on "leave", or how the lips purse on "food". Because these patterns are fine-grained and often similar across people, prior talking head methods that learn average motion from a dataset tend to produce overly uniform facial motion.

TalkLikeYou's response is to model habit in motion space rather than image space. A motion extractor turns the source image and a reference video into implicit keypoint-based representations, decomposed into canonical keypoints, expression deformation, and pose parameters (rotation, translation, scales). The generator then predicts habit-aware rotation and expression deformation, which are recombined into habit-aware driven keypoints before rendering. Because the implicit keypoints are semantically tied to facial components, the authors note that lip-related keypoints can be selectively edited for video dubbing.

One sampling step makes real-time generation possible

The efficiency claim rests on the training objective. Conventional Flow Matching learns a velocity field that transports noisy samples toward the data distribution, but the authors observe that directly predicting the velocity field introduces noticeable jitter in generated motion. They instead reparameterize the model to predict the clean motion at each timestep, which they report improves motion coherence and enables one-step sampling during inference.

The generator is a DiT with four progressive layers: self-attention extracts intrinsic motion representations, audio cross-attention adds synchronization, a habit-injection layer fuses either a reference habit sequence or a preset one-hot habit code, and a temporal attention layer refines sequence-level dynamics. Training combines an L2 reconstruction loss between predicted and ground-truth motion with a first-frame loss that aligns the first generated frame to the last frame of the previous motion, keeping consecutive context windows seamless. This is the mechanism behind the real-time claim, not an independent benchmark of wall-clock speed.

Two-stage training aligns preset codes and reference videos

Users can specify a target habit in two ways: a preset style from the dataset, or a reference video. The paper's two-stage imitation learning strategy is what makes both routes share one space. In stage one, the model reconstructs the habit embedding and is trained using only the one-hot code, with an alignment loss between reference video features and their corresponding one-hot habit codes. In stage two, a large-scale video dataset is used as the habit reference.

A t-SNE visualization of the unified habit space shows preset one-hot features as stars and reference-video features as dots: reference features of the same identity cluster around their corresponding preset feature, while different identities form distinct clusters. The authors also report that interpolating between two points in this space produces a new speaking habit combining characteristics of both, which is how the framework supports blending multiple habit sources.

PLAD scores imitation and diversity on separate axes

Existing metrics were not designed to measure speaking-habit similarity and diversity, so the paper introduces PLAD. For a generated motion sequence, PLAD extracts the lip-region expression deformation per frame and projects it onto three representative articulation axes: mouth opening, lip widening, and lip pursing, corresponding to words such as "bus", "leave", and "food". Each projection sequence becomes a normalized histogram, and the PLAD descriptor is the set of these histograms.

Two comparisons follow from that descriptor. Habit consistency between two motion sequences is measured by histogram IoU and Jensen-Shannon distance: higher IoU and lower JS distance mean stronger habit consistency, while lower IoU and higher JS distance mean better habit separability. Diversity is a different quantity: the average pairwise JS distance over a group of N generated sequences, which the authors say reflects whether a model generates distinguishable speaking habits instead of collapsing to similar distributions. Imitation accuracy and diversity are therefore not interchangeable readings of the same number.

Different methods carry distinguishable implicit lip patterns

The diversity analysis uses videos generated by other methods as references. In PLAD space the four methods show distinct distributions: Sonic tends to produce stronger projections along the lip-pursing axis, which the authors suggest may partly explain its high lip-sync scores if the synchronization metric favors such articulation patterns; Hallo2 tends to generate wider mouth movements; FLOAT and Ditto produce smaller lip movements with more concentrated distributions.

The authors interpret these differences as each method implicitly learning characteristic lip-motion patterns, analogous to distinct speaking habits of individuals. The practical consequence they draw is that TalkLikeYou can take these generated videos as references and adapt to their motion distributions, so habit imitation is not limited to real-world human reference videos. This is the paper's own framing of a capability, not an independent test of method-to-method transfer.

User study favors habit imitation across five dimensions

A user study rated five dimensions: lip synchronization, image quality, temporal consistency, habit imitation, and overall realism. For each method the study used 10 generated video clips, each longer than 5 seconds, with a five-point Mean Opinion Score scale where higher is better and more than 20 participants. TalkLikeYou achieved the highest scores across all evaluated dimensions, and the authors highlight habit imitation in particular, reporting that it reproduces target speaking habits while maintaining lip synchronization, visual quality, and temporal consistency.

The scope of this evidence matters: the user study is a perceptual rating on a fixed clip set, while the PLAD results are histogram-based motion measurements. They support the same overall conclusion from different angles, but a high opinion score is not a PLAD value and should not be read as one.

Where habit imitation stops working

The authors state two limitations. The method may struggle to maintain motion stability for exaggerated articulation patterns, such as excessively wide mouth openings. And the framework focuses primarily on the mouth region, where speaking habits are readily observable, rather than modeling person-specific behavior across the entire face. They plan to explore eye movements and head pose dynamics to capture a broader range of individual speaking behaviors.

The paper also acknowledges ethical risk: because the generated videos imitate recognizable aspects of individual behavior, misuse on public platforms is a concern, and the authors commit to responsible dissemination, supporting deepfake detection, and promoting safe use of generative media. These boundaries sit next to the capability they qualify: the habit signal is a mouth-region signal, and stability is not guaranteed at the extreme end of articulation.

Choose the habit source that matches your goal

The most decision-relevant split in this paper is the habit source. A preset one-hot code gives a dataset style without needing a reference clip, while a reference video lets the model adapt to a specific person's motion distribution, including distributions taken from AI-generated videos rather than real humans. The two-stage training is what places both options in one unified habit space, and interpolation between points in that space is the route to blended habits.

The second split is what you are measuring. If the question is whether a generated motion matches a target habit, the relevant PLAD readings are histogram IoU and JS distance between two sequences. If the question is whether a model produces varied habits at all, the relevant reading is the average pairwise JS distance across N samples. Reading one as the other would misstate what the paper actually shows, and the reported stability limit on exaggerated articulation is the condition under which the imitation claim is weakest.

Sources used in this answer