How can multimodal semantics and temporal over-regularity detect AI-generated videos?

MTOR adds caption semantics to visual detection

Most AI-generated video detectors base their predictions primarily on visual representations, leaving caption-derived textual semantics largely unexplored. The paper's first empirical claim is that caption-derived textual representations provide complementary discriminative cues to global visual representations, and that combining the two consistently outperforms either representation alone.

This matters because the caption is generated from the input video itself: the captioning model produces a textual description without any externally provided text prompt, and the video remains the sole input. The multimodal branch therefore adds a semantic view of the same video rather than new information from outside it.

AI-generated videos look temporally over-regular

The paper's second finding is that AI-generated videos exhibit stronger temporal persistence and lower temporal variability than real videos, a pattern the authors term temporal over-regularity (TOR). This is a property of fine-grained visual representations over time, not of the video's semantic content.

The TOR component quantifies this pattern with six temporal descriptors computed from frozen video-encoder tokens, organized at three levels: coarse inter-frame continuity, fine-grained token correspondence, and frame-to-video stability. Coarse continuity averages tokens within each frame, normalizes the frame centroid, and takes the cosine similarity between consecutive frame centroids. Fine-grained correspondence matches normalized tokens between adjacent frames using a symmetric bidirectional maximum-similarity measure. Frame-to-video stability compares each frame's tokens against the concatenated tokens of all remaining frames.

For each level the descriptors capture both strength and variation: the mean and standard deviation of the continuity scores, the mean and standard deviation of the correspondence scores, the standard deviation of frame-to-video similarity, and the mean absolute change between consecutive frames. The claim is that AI-generated videos sit at unusually high persistence and unusually low variability on these measures.

The two branches are complementary, not redundant

The paper tests whether multimodal semantics and temporal over-regularity carry separate information by comparing three variants averaged over five benchmarks: GenVideo, EvalCrafter, VideoPhy, VidProM, and GenVidBench. The multimodal branch alone reached 97.26% AP, 97.17% AUC, and 89.51% ACC. The TOR-only variant reached 93.22% AP, 94.59% AUC, and 84.19% ACC.

Combining the two predictions raised performance to 97.76% AP, 97.92% AUC, and 90.24% ACC. The authors read these gains as evidence that temporal over-regularity provides discriminative cues beyond the global multimodal representation, and they implement the combination as score-level fusion.

The fusion is calibrated on source-training statistics only: each branch's score is standardized using the mean and standard deviation estimated on the source-training set, then the two standardized scores are averaged. The calibration statistics are fixed after training, and no target-domain samples are used during inference.

Five benchmarks and 46 generator variants

Training uses the official training split of GenVideo as the source dataset, divided into source-training and source-validation subsets. Real videos are sampled from Youku-mPLUG, and synthetic videos are generated by Pika. Twenty percent of the source data is reserved as a stratified validation set for learning-rate scheduling, early stopping, and hyperparameter selection.

Testing collects synthetic videos from 46 generator variants across five benchmarks, with real videos uniformly sampled from MSR-VTT following GenVideo's standard setting. The generator families named in the packet include Gen2, Sora, ModelScope, MoonValley, MorphStudio, Show-1, HotShot, LaVie, Crafter, and WildScrape in GenVideo, and additional variants such as VideoCrafter V0.9, Gen2-December, MixSR, HotShot-XL, and LaVie-Interpolation in EvalCrafter.

The paper reports state-of-the-art overall performance against 16 representative baselines across these benchmarks. Because the headline numbers are means over five benchmarks, they describe average behavior across that benchmark set rather than performance on any single generator.

Detection holds under twelve real-world perturbations

The robustness evaluation compares MTOR with four representative baselines under 12 perturbation settings on GenVideo, covering H.264 compression at QP 23, 30, and 38; H.265 compression at QP 28, 34, and 40; resolution scaling; frame-rate conversion; color-space transformation; social-media simulation; TikTok stylized rendering; and TikTok heavy post-processing. These include both individual transformations and compound processing pipelines.

MTOR achieved the highest AP under all 12 settings, with an average AP of 98.79% and a range from 98.60% to 98.94%, a variation of only 0.34 percentage points. The strongest baseline, D3, averaged 90.54% AP, 8.25 percentage points lower. DeCoF, ReStraV, and FID averaged 90.39%, 87.91%, and 70.98% AP respectively.

The stability gap is as notable as the average gap. AP ranges were 2.30 points for D3, 15.77 for DeCoF, 16.38 for ReStraV, and 26.94 for FID, against 0.34 points for MTOR. DeCoF reached 96.28% AP under resolution scaling but fell to 80.51% under frame-rate conversion and 81.01% under TikTok heavy post-processing; FID fell to 56.04% AP under color-space transformation. MTOR stayed above 98.6% AP in every evaluated perturbation.

Generator subsets differ in how cleanly they separate

A t-SNE visualization on the 10 GenVideo generator subsets concatenates the multimodal branch's last hidden-layer representation with the standardized TOR descriptors and projects the result into two dimensions. The paper notes that this concatenation is used only for visualization, while the final MTOR prediction comes from score-level fusion.

Real and AI-generated samples generally form distinguishable distributions, but the degree of overlap varies by generator. Clear class separation appears on Gen2, Crafter, HotShot, ModelScope, MoonValley, MorphStudio, and Sora, while LaVie, Show-1, and WildScrape show relatively more overlap near class boundaries. The authors state that these separation patterns are consistent with the high AP and AUC obtained on those subsets.

Watch per-generator behavior, not just the average

The most decision-relevant reading of this paper is a comparison between two kinds of evidence. The averaged numbers — 97.76% AP, 97.92% AUC, and 90.24% ACC across five benchmarks, and 98.79% average AP across 12 perturbations — describe MTOR's typical behavior. The per-generator picture is more uneven: LaVie, Show-1, and WildScrape are the subsets where real and synthetic samples overlap most near class boundaries.

For a reader deciding whether this detection paradigm transfers to a specific deployment, the paper's own variables point to the questions that matter. Which generator produced the videos in question, and does it fall among the cleanly separated families or the overlapping ones? Will the videos pass through compression, resolution or frame-rate changes, color-space transformation, or social-media and TikTok-style processing, given that MTOR's AP range across those settings was 0.34 percentage points while baselines ranged from 2.30 to 26.94 points?

The paper's own framing also sets a boundary worth keeping in view: the reported gains come from score-level fusion calibrated on source-training statistics with no target-domain samples at inference, and the headline results are means over five benchmarks covering 46 generator variants. A single-generator deployment is therefore a narrower question than the one the averages answer.

Sources used in this answer