How does HENet++ achieve multi-task 3D perception and end-to-end driving?

Why large encoders and long history collide

The paper starts from a practical tension in camera-based 3D perception. Large image encoders, high-resolution inputs and long temporal context each improve representation quality, but using all three at once is expensive in both training and inference. A second tension sits on top: 3D object detection, BEV semantic segmentation and occupancy prediction want different feature representations, so a single shared backbone tends to trail specialized single-task systems.

HENet++ treats both tensions as design constraints rather than tuning problems. Instead of scaling one encoder across every frame, it assigns different encoders to different temporal roles, and instead of forcing one feature type to serve every task, it keeps two complementary feature streams alive through the network. The rest of the paper measures whether that split preserves accuracy while lowering cost.

A large encoder near, a lightweight encoder far

The hybrid image encoding strategy is the paper's central mechanism. Short-term frames, which carry the fine spatial detail needed to localize objects, pass through a large encoder. Long-term temporal context, which mainly supplies motion and occlusion cues, passes through a lightweight encoder. This lets the model keep high-resolution spatial information and long history at the same time without paying large-encoder cost on every historical frame.

The ablation isolates why this matters. Compared with simply raising image resolution, hybrid encoding reached higher accuracy with faster inference and lower training cost. Compared with simply lengthening the temporal window, it reached higher accuracy while reducing training cost. The paper also reports that naive ensembling of the two baseline detectors with non-maximum suppression degraded performance, because the weaker BEVDepth4D introduced additional false positives — so the gain comes from the encoding design, not from combining outputs after the fact.

The same comparison holds beyond one detection family: improvements were also observed when comparing SparseBEV with HENet++, which the authors present as evidence that hybrid encoding benefits both BEV-based and sparse-feature-based perception pipelines.

Dense background and sparse foreground serve different tasks

The second mechanism is the joint use of dense and sparse features. Dense background features, guided by occupancy prediction, describe the scene broadly; sparse object-centric features from 3D detection describe foreground instances precisely. The paper argues that purely sparse representations focus on foreground elements and may miss scene context, while purely dense representations need extra processing to extract object instances and can introduce errors in that step.

Keeping both streams gives each downstream task a more suitable input and, in the authors' account, reduces cumulative errors before prediction and planning. This is a design claim supported by the multi-task results rather than a separately isolated causal test, so it should be read as the paper's explanation of why the architecture helps.

Multi-task costs little against single-task training

The most decision-relevant compatibility result is how much accuracy the joint model gives up. Moving from single-task training to the end-to-end multi-task setting cost 0.4 mAP for 3D object detection, 0.5 mIoU for BEV semantic segmentation and 0.8 mIoU for occupancy prediction on the nuScenes validation set. Those are small drops for a model that now serves three perception tasks plus prediction and planning.

Occupancy results are reported on the nuScenes validation set, where HENet++ reaches state-of-the-art performance against its primary baseline FB-Occ; the authors also report RayIoU alongside mIoU as a complementary measure less sensitive to thickness and visibility inflation. The two metrics answer different questions and should not be substituted for one another.

Merged single-task weights beat single-task initialization

HENet++ adds a model-merging pretraining step: weights from multiple single-task models are merged to initialize the multi-task model. On a triple-task ResNet-50 model repeated five times with different random seeds, merged initialization reached 53.8 ± 0.3 NDS, 55.3 ± 0.3 BEV mIoU and 37.6 ± 0.2 occupancy mIoU. Initializing from detection weights gave 53.6 ± 0.2, 54.3 ± 0.5 and 36.2 ± 0.3; segmentation weights gave 52.8 ± 0.4, 55.0 ± 0.3 and 34.9 ± 0.6; occupancy weights gave 53.2 ± 0.2, 54.9 ± 0.5 and 37.3 ± 0.2.

The authors interpret this as a more balanced starting point that mitigates the task bias introduced by single-task pretraining. The merging coefficient α is not a sensitive knob: performance across 0.7 to 0.9 was comparable and within the standard deviation range, so the reported gain does not hinge on fine-tuning that hyperparameter.

Planning gains come from combined supervision

The end-to-end driving model sits on the perception framework and supports both vision-only and radar-camera inputs. Its loss ablation shows that supervision choices, not just architecture, drive safety. With only the planning loss, mean L2 was 1.69 and mean collision rate 0.52 on UniAD metrics, and 0.71 and 0.17 on VAD/STP3 metrics. Adding the prediction loss moved those to 1.36 and 0.20, and 0.58 and 0.10. Adding the collision-aware loss brought them to 1.29 and 0.13, and 0.55 and 0.05.

The two metric families are reported separately because they use different protocols; the paper does not claim they are interchangeable. The pattern within each family is consistent: each added objective improves trajectory quality and reduces unsafe behavior, which the authors read as complementary supervision for stable end-to-end learning.

Radar input and the collision-rate gap

The headline planning comparison is a collision rate, and the input configuration changes it. On the UniAD metrics, HENet++ with camera only averaged 0.17% collision rate and 1.44 m L2; adding radar brought the average to 0.13% and 1.29 m. On the VAD/STP3 metrics, camera only gave 0.06% and 0.61 m, radar-camera gave 0.05% and 0.55 m. For comparison, SparseDrive reported 0.32% and MomAD 0.30% on UniAD metrics, and SSR 0.06% on VAD/STP3 metrics.

The paper also reports higher PDMS than existing end-to-end driving models on the NAVSIM benchmark, described there as open-loop results with closed-loop metrics. The practical reading is that the radar-camera configuration is the stronger one in this paper's own tables, and that collision rate and L2 should be read together rather than as a single safety score.

Where the claim holds and where it stops

All perception and planning numbers come from nuScenes, with planning additionally evaluated on NAVSIM; the nuScenes benchmark covers 1,000 scenes split 700 for training, 150 for validation and 150 for testing across urban, highway and rural environments. The end-to-end system was fine-tuned for 10 epochs without CBGS after the prediction and planning decoder was attached, and the collision threshold in the collision loss was set to 3 meters — conditions a reader should keep in mind when comparing against systems trained differently.

The paper's own framing also bounds the multi-task claim: the small single-task-to-multi-task drops are measured on nuScenes validation, and the authors present hybrid encoding as compatible with diverse 3D feature extraction pipelines rather than as a universal replacement. Within those bounds, the priority the evidence supports is clear: if the goal is one model covering detection, segmentation, occupancy and planning, the hybrid-encoding plus dense-sparse design is the paper's answer, and the radar-camera variant is the configuration to compare against vision-only baselines on collision rate and L2 together.

Sources used in this answer