Can evidence sufficiency learning improve selective multimodal rumor detection?

Relevant evidence is not the same as sufficient evidence

Multimodal rumor detection here means image-text veracity assessment: deciding whether a caption's claim about when, where, or why an event occurred is false. The paper's starting observation is that retrieved evidence can be topically relevant yet still fail to settle that claim. Pages may repeat the claim under investigation, omit decisive details, or disagree about the event, and a count of matching pages is especially misleading when several pages copy a single source.

SAFE-MR therefore treats evidence sufficiency as a separate target. Alongside the binary veracity label y, the framework defines a target a*(x, E) in {0,1} indicating whether the supplied, admissible evidence permits a justified decision. The action space is {0,1, abstain}, and abstention is explicitly not a third truth category. Annotators judge sufficiency from the supplied evidence packet rather than from unrestricted web access or remembered facts, and unresolved conflict is recorded separately: disagreement from a demonstrably derivative or unreliable source need not force abstention.

Atomic claims and provenance-aware aggregation carry the method

A vision-language backbone extracts atomic propositions grounded in the post, each recording a predicate, entities, temporal and spatial qualifiers, whether it is asserted by text or inferred from visual content, and a materiality flag indicating whether contradicting it changes the post's contextual interpretation. Non-visible dates and locations are not treated as directly observed image facts, and OCR is retained as a noisy observation with provenance.

These claims and the retrieved evidence form a relation-aware graph. A learned relation predictor assigns support, refute, or irrelevant over candidate claim-evidence pairs, while a reliability gate combines provenance, independence, temporal compatibility, and missingness features. Agreement is computed across provenance groups, so copied or syndicated reports do not become independent corroboration. Aggregation then weights evidence contributions by relation and reliability, with an explicit empty-evidence embedding when nothing is retrieved and the residual claim representation retained.

Two heads turn sufficiency into an abstention decision

Materiality-aware pooling produces a post-level representation that is concatenated with claim coverage, decisive-refutation, conflict, and provenance-independence summaries. Two heads then predict veracity and sufficiency: s estimates the probability that the supplied evidence permits a justified decision, and the paper states it is not defined as the number of retrieved pages or as maximum class probability.

At inference, a positive calibration temperature produces calibrated veracity probabilities and normalized uncertainty. The model returns a veracity decision only when sufficiency is at or above one threshold and calibrated uncertainty is at or below another; otherwise it abstains. Temperature is fitted on a held-out calibration partition, and decision thresholds are selected on a separate tuning partition using temperature-calibrated uncertainty. The paper notes that calibration learned in-domain is not assumed to guarantee calibration after distribution shift.

Intervention training rewards stability and sensitivity separately

The training objective combines veracity, stance, and sufficiency supervision with an intervention term and a Brier proper scoring loss; missing auxiliary labels are masked rather than imputed as negative. Interventions are constructed on training evidence, with analogous perturbations evaluated separately on held-out examples.

Irrelevant addition preserves label and sufficiency when annotation confirms no material information is introduced, and prediction drift is penalized with Jensen-Shannon divergence and squared sufficiency differences. Key deletion lowers the sufficiency target only when all redundant decisive evidence is removed and the remaining packet becomes insufficient, using a ranking margin of 0.2. Conflicting addition receives packet-specific sufficiency labels, and pollution adds one group containing misleading evidence to a locally controlled packet. Crucially, these interventions do not require the veracity prediction to flip toward false evidence: the truth label stays fixed while acceptance should decrease when justification is lost.

Macro-F1 gains appear on all three benchmarks

SAFE-MR is evaluated on NewsCLIPpings, VERITE-full, and XFacta. It reaches macro-F1 scores of 91.2%, 75.8%, and 85.2% respectively. Against the matched backbone with evidence, the gains are 2.2, 4.9, and 4.8 percentage points.

The evaluation populations differ in ways that matter for reading these numbers. NewsCLIPpings provides synthetic out-of-context pairs, and this study uses 100,000 training, 5,000 validation, and 5,000 test examples with no exclusions, each split balanced 50% between authentic matches and out-of-context mismatches; the paper notes these counts describe the split used here rather than a universal official split. VERITE-full uses all 1,000 examples with binary labels, 338 truthful and 662 misleading, with both misleading categories mapped to false. XFacta contains 2,400 examples, an official development set of 240 (120 true, 120 false) and an official test set of 2,160 (1,080 true, 1,080 false), and covers image manipulation, image-text mismatch, and misleading text rather than out-of-context cases alone.

Selection error falls most where abstention matters

The diagnostic suite evaluates 600 base posts, 200 from each benchmark, under nine conditions: clean evidence; evidence deletion at 25%, 50%, and 75%; removal of all external evidence; irrelevant evidence addition; hard distractors; conflicting evidence; and evidence pollution. All 5,400 resulting packets are evaluated and manually labeled for sufficiency, stance, key evidence, and conflict.

On this diagnostic selection set, SAFE-MR reduces AURC from 0.105 for maximum-probability rejection to 0.075, and lowers error at 80% coverage from 13.8% to 8.5%. The paper frames coverage-matched comparison as the point: improvements over a forced classifier alone would not establish that explicit sufficiency learning is useful, and coverage matching avoids rewarding a model merely for declining more examples. Ablations associate sufficiency learning primarily with better selection and intervention training with greater perturbation robustness.

What the sufficiency labels and scope do not cover

Sufficiency targets come from GPT-5.6-Sol initial labels followed by individual human review. Two annotators perform first reviews, with a random 600 packets (20% of the 3,000-packet model-development set) second-reviewed by the other annotator without access to the first human answer; pre-adjudication agreement on binary sufficiency is 82%, and of 108 disagreements, 36 are resolved through subsequent annotator review and 72 are adjudicated by the principal investigator. The sufficiency target therefore depends on judgments of claim materiality and evidence admissibility, and combined model and human annotation can retain systematic ambiguities or biases, particularly for conflicting accounts.

Two scope conditions bound the results. VERITE posts do not enter model training, hyperparameter search, or temperature calibration, but 1,200 evidence packets derived from all 1,000 posts were used before protocol freezing to develop annotation rules, so VERITE is held out from model fitting yet was not untouched during annotation-protocol development. The paper also states that pretrained models may have encountered benchmark claims or fact-checking pages, that a downstream test split does not remove this possibility, and that the results do not establish temporal generalization or absence of pretraining contamination. Evaluation concerns static image-text verification; video manipulation, propagation modeling, and multilingual multi-image reasoning are outside its primary scope. Point estimates alone do not characterize run-to-run uncertainty, and synthetic evidence perturbations approximate only part of the missingness and disagreement encountered in deployment.

Judge the selector by coverage-matched error, not by abstention rate

The decision-relevant comparison in this paper is not the headline macro-F1 but the coverage-matched one. A detector that abstains more can look safer without being better, which is why the paper compares SAFE-MR against maximum-probability rejection and a learned selector at matched coverage, and why the reported AURC drop from 0.105 to 0.075 and the error drop at 80% coverage from 13.8% to 8.5% are the numbers that speak to selective verification.

The paper's own framing of when this matters is the split between irrelevant additions and decisive deletions. Intervention training is designed so that adding irrelevant evidence leaves predictions and sufficiency roughly unchanged, while removing all redundant decisive evidence lowers sufficiency without forcing the veracity label to flip. For readers assessing a selective rumor detector, that asymmetry is the practical test: check whether the system stays stable under harmless evidence growth and becomes less willing to decide when justification is actually lost, rather than whether it simply abstains more often.

Sources used in this answer