Gaze is stochastic, but not unstructured
Egocentric gaze prediction asks where a camera wearer is looking, using only the head-mounted video. The paper starts from a fact that limits any single-point predictor: even the same observer viewing the same scene does not reproduce the same trajectory exactly. Gaze is therefore better treated as a sample from a distribution over trajectories than as one correct answer per frame.
That randomness is not unconstrained. The paper identifies three joint influences: oculomotor dynamics that alternate saccades, fixations and microsaccades; top-down task goals, which the authors note are often revealed in future frames; and bottom-up visual saliency such as contrast and motion. GazeFlow's design choice follows directly from this: model the joint distribution of all gaze positions in a clip, conditioned on the whole video, rather than predicting each frame independently.
One ODE transport replaces per-frame prediction
GazeFlow is built on conditional flow matching. A learned velocity field takes a Gaussian noise sample and iteratively transports it into a plausible gaze trajectory, with each Euler integration step evaluating the velocity network once. Repeating the integration with different noise samples yields different trajectories, which is how the model expresses gaze stochasticity instead of collapsing to a single mode.
The velocity network receives two conditioning paths. Bottom-up features come from a V-JEPA 2 ViT-L video encoder, with frames resized to 256×256; top-down information comes from a task-token bank formed by pooling those features and querying them with learnable queries. Inside the network, six DiT-style blocks let gaze tokens exchange information across frames through time-aware self-attention with 1D rotary position embeddings, read same-frame visual tokens through per-frame cross-attention, and read the task tokens through gated cross-attention. The frame-to-frame self-attention is the only pathway where the joint structure across time enters the model.
Joint modeling versus four prediction paradigms
The paper's comparison table separates existing approaches by what distribution they model. Regression predicts one gaze point per frame with mean squared error and collapses gaze to a single mode. Marginal density estimation, as in GLC, predicts an independent per-frame heatmap and misses temporal coupling. Autoregression, as in Attention Transition and EgoM2P, chains per-frame conditionals on previously generated gaze, which captures temporal dependency but accumulates errors under self-rollout.
GazeFlow is placed in a fourth category, distribution mapping, which samples the full trajectory through one video-conditioned transport. The appendix makes the distinction formal: a factorized per-frame model has an irreducible error equal to the total correlation between gaze positions across time, which is zero only if gaze positions are conditionally independent given the video. Since human gaze clearly is not, even a perfect per-frame marginal model cannot match the joint trajectory distribution.
Accuracy and temporal alignment move together
On the standard datasets EGTEA Gaze+, Ego4D, the reported result is state-of-the-art performance on per-frame metrics, and the generated trajectories align better with human gaze temporal dynamics than prior methods. These are two different claims: the first concerns per-frame prediction accuracy, the second concerns how well the sampled trajectory follows the fixation-and-saccade structure over time. The paper's argument is that the joint formulation improves both at once, rather than trading one for the other.
The qualitative comparison on EGTEA Gaze+ illustrates the per-frame claim: GazeFlow's predicted heatmap is described as tightly localized on ground-truth gaze positions relative to AT, GLC and EgoM2P. The backbone choice is supported separately by a linear probe on frozen features, where V-JEPA 2 ViT-L at its native 256×256 resolution matches Qwen3-VL-8B in AUC (0.944) and exceeds its best layer in F1 (0.378 vs. 0.370) and precision (0.292 vs. 0.280), while a scale-matched 240×320 input degrades every metric.
Sampling many trajectories from one clip
Because the output is a distribution, one clip can produce many trajectories. The paper reports K=50 GazeFlow predictions per frame in its failure-case visualization, which shows how the sampled cloud behaves rather than committing to a single path. This is the practical meaning of the stochasticity claim: the model's output is a set of plausible gaze paths, not one deterministic guess.
Training uses 64-frame clips paired with per-frame gaze. For each clip the method samples a flow time and a noise sample, forms a linear interpolant toward a dataset trajectory, and regresses the velocity field onto the per-sample velocity. On EGTEA Gaze+, a per-frame validity mask removes blink frames from the loss without changing the minimizer on valid frames. At inference, clips longer than 64 frames are split into overlapping windows with stride 32, each encoded once, and predictions in the overlap region are linearly blended by frame index to keep transitions continuous.
Where the predictions break down
The reported failure mode is specific: clips containing saccades whose endpoint lies outside the camera's field of view. A large saccade can traverse tens of visual degrees in 30–80 ms, so at the 24–30 fps recording rate the eye can leave the captured frame within a single sample. Once the target is outside the field of view, the model has no visual evidence to ground a prediction, because it is conditioned solely on the recorded video. The predicted trajectories stay on visible objects while the eye tracker reports a gaze position clamped to the image boundary.
A second scope condition concerns what is being modeled. GazeFlow conditions on a fixed input video, so the head movement that produced that video is part of the conditioning. The recovered distribution is therefore of residual eye movement, not the full distribution that would arise across observers freely interacting with the same scene, where head movement itself varies across trials. Jointly modeling head and eye movement is left to future work.
Full-clip conditioning rules out online use
The most consequential boundary for deployment is temporal. GazeFlow's prediction is conditioned on the entire video sequence, which the authors state limits its applicability to scenarios requiring real-time gaze prediction from egocentric video, such as gaze-contingent rendering, where future frames are unavailable. The paper also trains a causal variant that sees only past and current frames; it enables online gaze prediction at some cost in accuracy.
The resulting priority is a scenario split rather than a single verdict. When the full clip is available and the goal is accurate, temporally realistic gaze trajectories, the joint distribution model is the paper's recommended direction, and its advantage over per-frame and autoregressive paradigms is the point of the comparison table. When frames must be processed online, the causal variant is the relevant configuration, and the accuracy cost is the price of that constraint. The off-frame saccade case sits outside both: no amount of trajectory modeling recovers a target the camera never recorded, which is why the authors point to a wider field of view or a saccade-aware model instead.
Sources used in this answer
GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction
GazeFlow models egocentric gaze as a video-conditioned joint distribution over trajectories using conditional flow matching, achieving state-of-the-art per-frame accuracy on EGTEA Gaze+ and Ego4D with better alignment to human gaze temporal dynamics.