Can retraining-free edge video analytics improve rare-class accuracy with spare compute?

Frozen VFM inference leaves GPU headroom that retraining consumes

Edge video analytics runs on servers with only a few energy-efficient GPUs, and live streams drift away from the data a lightweight CNN was trained on. Continuous learning (CL) answers that drift by periodically labeling, profiling, and retraining the deployed model, but those tasks compete with inference for the same device. The paper's GPU-time breakdown shows the cost: under single-stream serving, a frozen VFM spends only 6% to 47% of GPU time on inference, while CL systems consume nearly the entire GPU capacity. The same gap appears in memory, where the frozen VFM uses 1.5 GB against 3.7 GB for Ekya, 4.2 GB for AdaInf, 4.9 GB for RECL, and 4.8 GB for MOCHA.

That headroom matters because it is predictable. VFM inference runs inside a known latency envelope, so the slack in each inference window is known at runtime and can be spent on auxiliary work without violating the serving deadline. The paper's accuracy comparison makes the trade-off concrete: the frozen VFM holds 88.8% average accuracy at every concurrency level tested, without any online labeling, profiling, or retraining, while the CL baselines lose accuracy as inference demand grows and less capacity remains for retraining.

The catch is what that 88.8% average hides. The deployment streams are heavily skewed: Car and Person dominate every stream, while each of the four rare classes accounts for under 10% of annotated objects. Aggregate accuracy on such a distribution can stay high even when rare classes are frequently wrong, which is exactly what the per-class results show.

Rare-class errors concentrate in a few confusion pairs

In the single-stream setting, the frozen VFM reaches 95.5% accuracy on Person and 98.4% on Car, but only 66.8% on Bicycle, 67.2% on Motorcycle, 73.1% on Bus, and 53.8% on Truck. The errors are not spread evenly across labels. They cluster into a small number of dominant confusion pairs: Truck to Car accounts for 41.0% of Truck ground-truth samples and Bus to Car for 23.8% of Bus samples, while the reverse directions are far smaller, at 0.5% of Car samples predicted as Truck and 0.4% as Bus. Other rare classes have their own pairwise structure, such as 14.8% of Bicycle samples predicted as Motorcycle, and Motorcycle most often confused with Person (12.6%) and Car (10.7%).

This structure is what makes selective correction plausible. Because most predictions are already correct, improving rare-class accuracy does not require replacing or continuously retraining the whole inference model. The paper's argument is that computation can be aimed only at predictions likely to fall into these confusion pairs, recovering much of the lost rare-class accuracy at a fraction of the retraining cost.

The paper also separates two things readers could easily conflate. Average accuracy across all samples and rare-class accuracy are different measures with different denominators: the first is computed over the full skewed stream, the second only over samples whose ground-truth label is rare. A system can look strong on the first while remaining weak on the second, which is the gap DIALER targets.

Correction pipelines are built offline and run on idle cycles

DIALER pre-builds multi-stage correction pipelines for the dominant rare-to-common confusion pairs offline, using curated public repositories that contain rich samples of rare classes. This decoupling matters because CL depends on incoming frames for retraining, and rare objects appear too infrequently in live streams to provide effective supervision. The frozen VFM backbone maps both the offline samples and live frames into the same embedding space, so correction models prepared ahead of time can be applied to online predictions.

At runtime, DIALER routes correction candidates to the corresponding pipelines and executes as many stages as idle GPU headroom permits. The pipeline is ordered so that the earliest and cheapest work yields the largest accuracy gain, which is what makes partial correction useful when a budget runs out mid-sample. A lightweight adapter operates on cached VFM features, and costlier refinement stages run only when latency and memory budgets allow. Before dispatching a correction action, DIALER uses the remaining inference time and available GPU memory together with offline execution profiles to decide whether the action is predicted to fit; actions that no longer fit are pruned along with their dependent work, and affected samples keep their most recent valid predictions.

Two design constraints shape this scheduling. First, the system must decide from inference-time information alone which predictions warrant correction, since applying correction indiscriminately would waste resources on already-correct predictions and could flip correct common-class predictions. Second, refinement model weights are kept in pinned host memory with a bounded cache of GPU-resident copies, and non-resident models are prefetched asynchronously so that transfer overlaps with executable correction work.

The adapter carries most of the gain, deeper stages add little

When compute and memory suffice to run every eligible correction stage, the adapter alone raises Truck accuracy from 53.8% to 71.5%, and the two refinement stages add 0.7% and 0.3%. On the common classes, Car and Person lose 0.8% and 0.4% respectively, so the rare-class gains come with little degradation on the classes that dominate the stream. Across the four streams, average rare-class accuracy improves by 9.5% to 15.5%, with the adapter again providing most of the gain.

The diminishing return with correction depth is the property the paper identifies as necessary for partial correction to be useful. The cheapest stage recovers most of the accuracy, and each deeper stage adds less while covering harder samples that earlier stages could not fix. That ordering is what allows DIALER to skip deeper stages under a tight budget and still retain most of the accuracy gain.

The paper also compares DIALER against simpler correction designs under the same common-class preservation constraint. A prototype classifier improves rare-class accuracy by 3.4%, feature-space kNN by 4.5%, and a shared MLP by 6.0%, against 14.0% for DIALER. The paper describes this comparison as empirical support for its staged, confusion-pair-specific design rather than as a general proof of superiority.

Gains hold until per-stream headroom runs out

Co-locating more streams on a fixed server is how contention arises in deployment. The rare-class gain barely changes as concurrency increases up to thirteen streams, because the adapter continues to execute almost all of its eligible work while the deeper refinement stages are the first to be dropped. At fourteen streams, the remaining headroom can no longer sustain even the adapter, and the gain falls close to zero.

Latency behavior follows a similar threshold. The window miss rate, defined as the fraction of inference windows whose final prediction is produced after the 1 s deadline, stays at 1.0% or below through eight streams, then rises to 5.63% at ten, 10.67% at twelve, 42.88% at fourteen, and 97.88% at sixteen. These misses come from dynamic contention among co-located streams, since MPS does not isolate GPU resources between them. DIALER drops optional correction work before the deadline but cannot shorten the VFM inference that every stream must run, so at fourteen streams the serving path alone can no longer keep pace.

Varying the compute budget alone shows where correction starts to bind. Rare-class accuracy stays near 76.6% as the per-stream MPS allocation is reduced from 100% to 12%, and falls only slightly to roughly 75.6% at the most constrained 6% allocation, where execution of the refinement stages collapses but the adapter still executes most of its eligible work. With a dedicated allocation rather than MPS sharing, no window misses occur down to a 25% GPU share.

Correction transfers to detection and other backbones

The classification experiments use pre-cropped objects, but deployed pipelines typically run detection, where the same rare-class confusions appear: a truck that is localized correctly but recognized as a car is still a rare-class error. DIALER runs after a frozen DETR-based detector has completed its normal selection pipeline, so it changes neither which objects are detected nor where their boxes are; it revises only the predicted class of each detection, through the same admission criterion, adapter, and refinement stages.

On the original full frames of the online partitions, DIALER raises rare-class AP from 42.4% to 49.2%, with per-class gains of 5.4% for Bicycle, 9.7% for Motorcycle, 5.0% for Bus, and 7.1% for Truck. Because localization is left untouched, these gains come entirely from correcting recognition errors. The paper reports that the approach also remains effective across three VFM backbones and on a Jetson Orin NX, which extends the evidence beyond the single-GPU RTX 3090 setup used for the main measurements.

Where offline correction stops working

DIALER targets dominant confusion pairs that can be identified before deployment, so its advantage narrows when mispredictions spread evenly across many class pairs, when previously unseen classes appear, or when the deployment distribution drifts far enough to invalidate the current correction pipeline. The paper states that these are the cases where continuous learning still has a role, and frames combining periodic model updates with selective correction as complementary future work.

The evaluation also places all streams on a single GPU, where correction competes with inference for the same device. The paper notes that DIALER sustains correction far better under a dedicated allocation than under MPS sharing, and suggests two multi-GPU directions: partitioning streams across GPUs to mitigate per-GPU contention, or placing correction models on a separate device so that one stream's correction no longer contends with another stream's inference. Both would require changes to the budget-aware scheduler but not to the correction pipeline itself.

A further scope condition is that correction models are prepared offline from public data, so once constructed they can be applied to every stream that shares the VFM and a class set. The multi-stream experiments already reuse a single correction pipeline across co-located streams, which means a server can add a stream without repeating offline preparation, though sharing one copy of the models across MPS instances would need additional system support.

The priority is which errors to correct, not whether to retrain

The decision this paper supports is not whether to abandon retraining, but where to spend the compute that retraining-free inference frees. The relevant threshold is the shape of the error distribution: when rare-class mistakes concentrate in a few dominant rare-to-common pairs, staged correction recovers most of the loss, and the adapter alone captures the bulk of it. When mistakes are diffuse, or when the deployment distribution drifts far enough to invalidate the offline pipelines, the paper's own framing points back toward continuous learning.

The second threshold is headroom per stream. The rare-class gain holds up to thirteen co-located streams and collapses at fourteen, and window misses stay at or below 1.0% through eight streams before climbing sharply. For a reader deciding whether this approach applies, the paper's evidence suggests checking two things first: whether the rare-class errors in the target stream form a small number of dominant confusion pairs, and whether the serving configuration leaves enough per-stream GPU headroom for the adapter to keep executing. The adapter is the stage that carries the accuracy gain, so it is the stage whose timely execution rate determines whether the benefit survives contention.

Sources used in this answer