[CVPR 2026] VisPrompt: Anchoring Prompt Learning to Visual Truth to Defeat Label Noise

TL;DR

Prompt learning has revolutionized the adaptation of Vision-Language Models (VLMs), but it has a "glass jaw": it breaks easily under label noise. VisPrompt fixes this by shifting the guidance of prompt optimization from the often-corrupted text labels to the inherently reliable visual content of the image. By using cross-modal attention and gated modulation, it achieves superior robustness, maintaining high performance even when 75% of the training labels are wrong.

Problem: The Fragility of Soft Prompts

In the standard paradigm (like CoOp), we keep the VLM backbone (e.g., CLIP) frozen and only train a few "soft prompt" tokens. These tokens are like a sponge—they absorb information from the gradients to bridge the gap between the pre-trained model and a specific task.

However, when labels are noisy (e.g., an image of a "dog" labeled as "cat"), the prompt is forced to learn an incorrect mapping. Because the backbone is frozen, the prompt tokens have no choice but to "memorize" the noise to minimize the loss, leading to a catastrophic drop in generalization.

Methodology: Seeing is Believing

The authors' core insight is simple yet powerful: The image doesn't lie, even if the label does. Even if a sample is mislabeled, the image contains the correct semantic features.

1. Cross-Modal Visual Conditioning

Unlike previous methods that treat prompts as static or purely text-driven, VisPrompt uses Cross-modal Attention. It treats the learnable prompt tokens as "queries" to extract relevant "keys" and "values" from the local visual patches of the image. This "grounds" the prompt in actual visual evidence.

2. FiLM-based Robust Modulation

Not all visual cues are of equal quality. To prevent irrelevant visual information from corrupting the prompt, VisPrompt introduces a FiLM (Feature-wise Linear Modulation) gating mechanism.

• Scale and Shift: Adaptive parameters modulate the prompt embeddings.
• Gating: A sigmoid-based gate decides how much visual information should influence the prompt based on the current instance's reliability.

Figure: The VisPrompt architecture, showing how visual features are projected and injected into the text-side prompt through attention and FiLM gating.

Experiments: Superior Resilience

VisPrompt was tested against heavyweights like NLPrompt and JoAPR across seven datasets.

• Quantitative Dominance: On the OxfordPets dataset, VisPrompt maintains near-90% accuracy across almost all noise levels, while other methods plummet as noise increases.
• Extreme Noise: In 75% noise scenarios (where 3 out of 4 samples are wrong), VisPrompt remains functional, outperforming traditional robust methods by massive margins (up to 15% on EuroSAT).

Table: Accuracy comparison under Symmetric and Asymmetric noise. Note the stability of VisPrompt in high-noise columns (50% - 75%).

Why It Works: A Theoretical Perspective

The authors provide a theoretical proof (Theorem 1) demonstrating that the deviation of the noise-trained prompt from its "ideal" clean counterpart is bounded. By suppressing irrelevant visual tokens at the attention stage and using Lipschitz-continuous FiLM modulation, the framework ensures that even if the supervision is noisy, the update signal remains anchored to the stable visual manifold.

Conclusion & Insights

VisPrompt represents a pivot in the "Noisy Label Learning" field. Instead of just trying to "clean" the labels (which is difficult and error-prone), it utilizes the cross-modal redundancy of VLMs.

Takeaway for Practitioners: When adapting VLMs to messy, real-world data, don't just rely on your labels. Let the model's own visual understanding guide its learning process. The future of robust AI lies in this kind of self-correcting, multimodal feedback loop.

Limitations

• Computational Overhead: The FiLM block adds a small inference cost.
• Scope: Currently validated primarily on CLIP-style backbones and classification tasks.