[CVPR 2025 Prediction] MoD-DPO: Decoupling Modalities to Cure Omni-modal Hallucinations

TL;DR

Omni-modal Large Language Models (Omni LLMs) like Qwen2-Omni or MiniCPM-O can "see" and "hear," but they often suffer from cross-modal hallucinations—imagining a dog barking because they see a dog, even if the audio is actually a car horn. MoD-DPO solves this by fundamentally restructuring the Preference Optimization objective. It forces the model to ignore irrelevant noise while becoming hypersensitive to the correct sensory modality, effectively "decoupling" the internal reasoning pathways.

The "Linguistic Shortcut" Problem

Why do state-of-the-art models fail?

1. Over-reliance on Language Priors: The LLM backbone is so good at predicting text that it often ignores the video/audio pixels or waveforms.
2. Spurious Correlations: If the training data often pairs "beach video" with "wave sounds," the model creates a "shortcut" that makes it unable to handle a silent beach video or a beach video with heavy metal music.

Methodology: The Power of Invariance and Sensitivity

The core innovation is a "Modality-Decoupled" objective. Instead of standard DPO, which just compares a "good" vs. "bad" answer, MoD-DPO introduces two regularization terms:

1. Invariance (${\beta }_{inv}$): If we add noise to the irrelevant modality (e.g., blurring the video for an audio question), the model's output should stay the same.
2. Sensitivity (${\beta }_{sens}$): If we add noise to the relevant modality, the model's output must change. This forces the model to prove it is actually "paying attention" to the right source.

Figure: The MoD-DPO pipeline involves stage-based generation of hard-negative preferences and a decoupled optimization objective.

The Language-Prior Debiasing (LPD) Penalty

To stop the model from "guessing" based on the text prompt alone, the authors add a penalty term: $r_{LPD}=-\log {\pi }_{text}(y|x)$ This mathematically suppresses the rewards for answers that the model could have guessed without even looking at the video or hearing the audio.

Experimental Battleground: AVHBench & CMM

The authors tested MoD-DPO against heavy hitters like VideoLLaMA 2 and Qwen2.5 Omni.

• AVHBench: The model achieved a relative 27% improvement in Audiovisual Matching accuracy.
• CMM (Curse of Multi-Modalities): It showed superior perception accuracy and hallucination resistance.

Figure: Average accuracy across AVHBench and CMM benchmarks, showing MoD-DPO++ outperforming baseline DPO and OmniDPO.

Visual Evidence: Tracking Attention

Does it actually change the model's brain? The authors plotted the attention distribution. With MoD-DPO++, the percentage of total attention directed toward audiovisual tokens increased significantly compared to the reference model. This proves the "debiasing" actually worked—the model "looked" more at the pixels.

Figure: Visualizing the shift towards audiovisual tokens in the attention layers.

Critical Insight & Future Outlook

The brilliance of MoD-DPO is that it is objective-centric, not just data-centric. While others try to fix hallucinations with "better data," MoD-DPO changes the rules of the game (the Loss Function).

Limitations: The framework currently treats audio and video as separate entities to decouple them. Future work might need to explore how to apply this to "intertwined" tasks (like reading lips) where both modalities are equally relevant and cannot be easily perturbed.

Takeaway for Practitioners: If your multimodal model is "lazy" and over-relying on text, don't just add more data. Use a decoupled DPO approach to force modality-faithfulness.