When does SWIR imaging outperform RGB for object detection in adverse autonomous driving conditions?

RGB matches or beats SWIR in most conditions

The paper compares paired RGB and SWIR object detection on the RASMD dataset, covering four weather conditions and two real-time detection architectures, with several fine-tuning strategies scored against a unified ground truth. The headline result is that RGB demonstrates comparable or superior performance in most scenarios. That is an aggregate statement about the whole evaluated set, not a claim that RGB wins in every frame.

The weather-wise results sharpen this. RGB clearly dominates SWIR under cloudy conditions for similar configurations, and the gap narrows progressively through sunny and rainy conditions. Under snowy conditions the two sensors become very competitive, and the paper states that a clear inference is not possible there. Precision stays broadly competitive across all weather, ranging between 0.73 and 0.93, with a relative decline for snow in both cameras attributed to complex scenes and reduced contrast.

RF-DETR is the more robust architecture choice

Architecture matters as much as modality here. RF-DETR outperforms the CNN-based YOLOv8 across modalities, weather conditions, and training strategies, and is described as the more robust framework. Within RF-DETR, all fine-tunings give comparable recall across the four weather conditions, whereas YOLOv8 degrades under weather-specific mix and occasionally weather-specific base configurations. The paper attributes this to YOLO being more susceptible to overfitting under limited per-condition training data.

Fine-tuning generally improves detection for adverse weather, and the improvement is more significant for the SWIR domain. Recall improves with fine-tuning across both models, both modalities, and all weather conditions, except for RGB under sunny and cloudy conditions, where fine-tuned models score comparably to the pretrained model because the visible spectrum aligns closely with the pretraining domain under good visibility. Weather-specific fine-tuning is the unstable option, producing sub-optimal YOLOv8 results and high fluctuation for RF-DETR.

Mixed fine-tuning with equal proportions of RGB and SWIR emerges as an effective single-model solution, encouraging modality-invariant representations and staying consistently close to the top, even achieving the best mAP under snowy conditions with the SWIR camera using YOLOv8. Synthetic SWIR augmentation also performs at par with SWIR-only trained models across all weather conditions, which the authors present as a promising direction for handling SWIR data scarcity.

A consensus framework finds rare SWIR wins without labels

Aggregate metrics can hide rare but safety-critical cases, so the paper proposes a sensor-dominance mining framework that sorts RGB-SWIR image pairs into RGB+, SWIR+, Equal, and Discarded categories using largely unannotated paired data. Three model groups act as independent voters: RF-DETR-Mix and YOLO-Mix, each processing both sensor images, plus YOLO-RGB and YOLO-SWIR processing their matching sensor image. Two detections match when they share a class label and their bounding boxes reach IoU of at least 0.3.

A group votes SWIR+ when it finds at least one exclusively SWIR detection with no RGB counterpart while all remaining detections match, and RGB+ in the mirror case. A vote where both modalities produce unmatched detections is discarded regardless of consensus, because no single modality can be declared dominant. The three variants M1, M2, and M3 then apply progressively stricter consensus rules, moving from high recall to high precision.

The distribution across methods shows RGB+ pairs consistently outnumbering SWIR+ pairs, reinforcing the aggregate finding that visible-spectrum imaging remains dominant or equal in most scenarios. The framework's stated limitation is that no reliable decision can be made for 61.5% of pairs, either because both modalities produce unmatched detections or because voter groups disagree. That discarded share is the price of operating without ground truth.

Manual curation reduces the SWIR set to 532 verified pairs

Because all three automated methods remain susceptible to false positives from model hallucinations and detection errors, the authors apply manual curation to the SWIR+ set. A candidate pair is retained only if manual inspection confirms the SWIR advantage contains additional valid detections compared with RGB after removing false positives, and only if consecutive filtered samples within a sequence are separated by at least 10 frames to avoid over-representing near-identical frames from the same scenario.

This yields a curated set of 532 distinct scene image pairs, named RASMD-HiLS for Human in the Loop Segregation. Visual inspection groups these samples into four recurring environmental scenarios: glare on the windshield, water droplets on the windshield, objects hidden due to contrast, and distant vehicles. The scenarios are not mutually exclusive, and a single image may belong to multiple buckets. All selected frames are then manually annotated under a common ground truth for both modalities, producing a high-quality SWIR-beneficial dataset used for subsequent analysis and validation.

Four scenarios show a clear SWIR recall advantage

Validation uses RF-DETR-RGB and RF-DETR-SWIR on their respective modality images, both held out during filtering. The paper reports that the mined dataset captures common scenario-dependent SWIR advantages, and that the advantage is primarily driven by improved recall.

Under glare, RGB weighted recall is 0.3318 against SWIR's 0.7204, and weighted F-1 is 0.4746 against 0.8076. This scenario corresponds to extreme visible-light saturation, typically from direct sunlight reflections when transitioning from low to high illumination, such as just before exiting a tunnel; the saturation degrades RGB image quality and causes missed detections while SWIR remains largely unaffected.

Under water on the windshield, RGB weighted recall is 0.6459 against SWIR's 0.8000, with weighted F-1 of 0.7465 against 0.8641. Scattering from droplets prevents RGB detections while vehicles are detected in SWIR. For hidden objects, RGB weighted recall is 0.7631 against SWIR's 0.8062, and the class-wise AP for person rises from 0.1380 to 0.2978. For very-far vehicles, RGB weighted recall is 0.7288 against SWIR's 0.7887, with class-wise AP for car rising from 0.5017 to 0.5753 and for truck from 0.5088 to 0.5541.

Precision stays comparable in these subsets, which is what makes the recall gains meaningful: the paper describes SWIR as improving recall while maintaining comparable precision. The authors also note that the mined SWIR+ subsets inherit the class distribution of the underlying dataset, where conditions such as glare are dominated by certain classes such as cars, which is why weighted or binary metrics are used and only statistically significant classes are validated.

Saturation, scattering, and contrast explain the split

The four scenarios share a common structure: RGB fails for a physical reason that SWIR is comparatively immune to. Glare saturates the visible sensor, water droplets scatter visible light, low-contrast scenes fall below the visible sensor's discrimination threshold, and long range suffers from atmospheric scattering that is reduced in SWIR. In each case the failure is a property of the visible band under that condition, not a general deficiency of RGB.

This is why the aggregate and scenario-level results are not contradictory. Aggregate metrics average over conditions where RGB is strong, including cloudy weather with high visibility and strong contrast, and those conditions dominate the dataset. The mining framework deliberately searches the tail where the physical mechanism flips. The paper frames the conclusion accordingly: SWIR should be viewed as a complementary modality that enhances perception in rare but challenging conditions, not as a primary replacement.

Where these results do and do not transfer

Several limits qualify the findings. The dataset reflects real-world driving conditions but exhibits strong class imbalance, and combined with sequential frames this makes sufficient scene diversity and class representation during training hard to ensure, which can lead to overfitting for underrepresented classes such as person AP. The mining framework operates without ground truth and relies on multi-model agreement, which may bias results toward model-specific behaviors, and a significant portion of data is discarded during filtering, potentially overlooking rare scenarios where SWIR could provide additional benefits.

The curated SWIR+ subsets also inherit the class distribution of the underlying dataset, so certain conditions are dominated by certain classes, which is why the validation restricts itself to weighted or binary metrics and statistically significant classes. The paper's own forward-looking suggestions, adaptive fusion strategies that dynamically leverage SWIR under challenging conditions and improved synthetic-to-real alignment for SWIR image generation, are stated as future work rather than demonstrated results.

Treat SWIR as a targeted complement, not a replacement

The decision-relevant reading of this paper is a priority order rather than a sensor verdict. For general road-scene detection across the four evaluated weather conditions, RGB remains the primary and reliable modality, and RF-DETR with mixed fine-tuning is the well-balanced cross-modal choice on speed, accuracy, and resources. SWIR earns its place specifically in the four mined scenarios, and the paper singles out two as most consequential: glare represents the most severe RGB failure, while low-contrast conditions mainly impact pedestrian detection, making SWIR particularly valuable for vulnerable road user safety.

That ordering is the paper's own emphasis, and it follows from the measured gaps rather than from a general preference for one sensor. A reader deciding where SWIR adds value should look first at glare and low-contrast pedestrian scenes, then at windshield water droplets and long-range vehicle detection, and should treat the 61.5% discarded share as the boundary of what this unlabeled mining approach could resolve.

Sources used in this answer