Can physically based synthetic data enable dense illuminant estimation?

Why dense illuminant labels are hard to get

Illuminant estimation recovers the chromaticity of a scene's light sources so that color shifts from varying lighting can be corrected. Learning-based models, including convolutional and transformer architectures, improve robustness by using contextual and semantic information, but their progress is constrained by the limited availability of large-scale datasets with accurate illuminant ground-truth. Acquiring such ground-truth requires controlled capture conditions and reference measurements.

The problem is sharper for multi-illuminant estimation, where illumination varies spatially across the scene and pixel-level annotations are required. Real images make this prohibitively difficult to obtain at scale, which is the gap this paper targets.

Reusing rendered scenes as illuminant supervision

The paper proposes a general, reusable pipeline that derives dense illuminant chromaticity maps from physically based 3D-rendered scenes. It repurposes an existing 3D scene collection, Hypersim, which provides geometric and photometric metadata and decomposes scene radiance into diffuse illumination and residual non-diffuse components. Because physically based rendering keeps illumination, reflectance, and observed color photometrically consistent, dense ground-truth illuminant maps can be derived from the rendering outputs.

Each rendered image is reconstructed as I = (R ⊙ L) + ρ, where L is diffuse illumination, R is diffuse reflectance, and ρ captures specular reflections, interreflections, and other non-Lambertian effects. The authors use L as a proxy for the scene illuminant, which is accurate for Lambertian surfaces.

Why diffuse illumination alone is insufficient

Relying on L alone fails in regions dominated by specular reflections, shadows, or other non-diffuse effects, where the residual term ρ contributes most of the observed radiance and L may contain near-zero or missing values rendered as black pixels. Direct use of L would produce incomplete or physically inconsistent illuminant maps.

To address this, the pipeline augments diffuse illumination with the grayscale intensity of the residual before normalization: L̂ = normalize(L + avg(ρ)), where avg(ρ) is the per-pixel mean over the RGB channels of the residual image. The residual is assumed to be dominated by view-dependent reflectance and thus provides limited reliable chromatic information, so this choice is intentionally conservative about chromaticity: it avoids injecting spurious color while preserving illumination magnitude in non-diffuse regions. The combined illumination is then normalized into dense per-pixel chromaticity maps.

Images are provided in HDR; the authors apply the dataset authors' tonemapping function but omit gamma correction to preserve a linear relationship between radiance and pixel values. Where the diffuse illumination image contains black pixels, they assume neutral illumination and generate binary masks to exclude those pixels from the training loss. A manual inspection removes malformed images, leaving a final set of 74,321 images, referred to as Hypersim-WB. The split follows a per-scene strategy to avoid data leakage: 80% of scenes for training, 20% for testing, with an additional 20% of the training set reserved for validation.

How synthetic pre-training was tested

All experiments use a consistent setup designed to isolate the contribution of synthetic pre-training from architectural choices, focusing on data efficiency, robustness under limited supervision, and convergence behavior rather than absolute peak performance. Models are first pre-trained on Hypersim-WB and then fine-tuned on real-world datasets, using the Adam optimizer with an initial learning rate of 1×10⁻⁴, cosine annealing with warm restarts every 10 epochs, and early stopping with a patience of 50 epochs during fine-tuning. Test images are resized to 256×256 pixels for inference and model outputs are resized back to original dimensions. Training uses an angular error loss, masking pixels where only the residual ρ contributes to radiance. Evaluation reports the angular error distribution: mean, best-quartile (B-25), worst-quartile (W-25), and 95th percentile (95-P).

For multi-illuminant estimation, the authors evaluate two image-to-image architectures, UNet and HDRNet, trained to infer dense illuminant maps from RGB inputs, and compare against Gray World and the FC4 model as statistical and learning-based single-illuminant baselines whose global predictions are expanded to the size of the ground-truth dense map. Learning-based models are pre-trained on Hypersim-WB and fine-tuned on the real-world LSMI and MIMO datasets, and are also trained from scratch on the target datasets for comparison.

Pre-training gains are largest when real data are scarce

Models pre-trained on Hypersim-WB consistently outperform those trained solely on real data. The improvement is particularly pronounced for smaller datasets like MIMO, which has 78 images, highlighting the role of synthetic pre-training in generalization and convergence. Even for larger datasets such as LSMI, pre-trained models maintain superior performance over their from-scratch counterparts.

A dataset reduction study on MIMO makes the data-scarcity effect explicit. The training set is progressively reduced to 1/2, 1/4, and 1/8 of its original size while the test set stays constant, using cross-fold validation and the UNet architecture, which was the top-performing model in earlier experiments. Performance degrades as dataset size decreases, but fine-tuned models consistently outperform from-scratch counterparts, and pre-trained models degrade much more slowly. Across the paper's experiments, synthetic pre-training yields gains of up to 28% for single-illuminant estimation and up to 57% for multi-illuminant estimation.

One dataset serves both estimation settings

Although Hypersim-WB is designed for multi-illuminant estimation with dense maps, it can be repurposed for single-illuminant tasks by averaging the ground-truth illuminant chromaticity over each image. The authors evaluate Gray World, FC4, and the UNet and HDRNet architectures, averaging their dense predictions to obtain a single RGB illuminant vector, with all models pre-trained using the same hyperparameters and fine-tuned on the ColorChecker dataset.

This matters because the same synthetic supervision supports two different estimation settings with different output granularity: a single global illuminant vector per image versus a dense per-pixel map. The reported gains should be read separately for each setting rather than treated as one interchangeable number.

Where the pipeline transfers and where it does not

The authors emphasize that the Hypersim instantiation serves as a proof of scalability rather than a dataset-specific contribution: the methodology itself is readily transferable to other rendering environments and synthetic scene collections, and it is independent of network architecture. The paper's future work points to applying the pipeline to datasets with outdoor and more complex lighting conditions, and to investigating domain adaptation to better transfer learned illuminant representations to real-world images.

The evidence here is bounded by the evaluated benchmarks and architectures. The pipeline's residual handling assumes the residual is dominated by view-dependent reflectance and provides limited reliable chromatic information, and black-pixel regions are handled by assuming neutral illumination and masking them from the loss. These choices qualify how the dense maps should be interpreted, especially in specular, shadowed, or non-diffuse regions.

What to watch when real labels are limited

The paper's most decision-relevant pattern is a scenario split rather than a single headline number: synthetic pre-training helps across datasets and tasks, but the advantage is strongest in multi-illuminant estimation and under limited real-data conditions, where pre-trained models degrade much more slowly as the real training set shrinks to 1/2, 1/4, and 1/8 of MIMO. For readers weighing whether to invest in synthetic pre-training, the relevant question is not whether it beats training from scratch on a large real benchmark, but how much real dense annotation can be replaced before the gap widens.

The practical threshold implied by this study is the availability of accurate dense illuminant ground-truth: when pixel-level real annotations are scarce or spatially varying illumination must be captured, the physically based synthetic pipeline is the more scalable starting point, with the caveat that its residual and black-pixel assumptions shape the supervision it produces.

Sources used in this answer