How can sparse convolutions make temporal action detection faster without losing accuracy?

The bottleneck dense adapters leave untouched

Adapter-based temporal action detection freezes a billion-parameter Vision Transformer backbone and trains small convolutional modules inserted into it. That keeps training cheap, but the adapters still read a dense token grid, so their cost grows with the full video sequence. The paper's starting point is that this leaves inference-time computation, and therefore scalability with video length, largely unaddressed.

Token selection is the natural remedy: it prunes uninformative tokens and lowers the quadratic cost of self-attention. The problem is structural. Convolutional adapters reshape tokens into a spatial feature map and apply grid-based convolutions, so removing tokens breaks the fixed neighborhood layout they assume. The usual fix scatters kept tokens back into a zero-filled dense tensor, convolves, then gathers outputs, and that reconstruction consumes much of the saving pruning created.

A convolution needs neighbors, not the whole grid

The paper's key observation is that a 3x3 convolution only needs nine neighbors per output position, not the entire spatial grid. If those adjacencies are encoded explicitly over the sparse token set, the grid becomes unnecessary. This is the idea behind native sparse 2D convolutions, the primitive the authors call SparseConv2D.

Implementation is a neighbor index table of shape N_kept x 9 that maps each kept token to the sparse-list positions of its spatial neighbors, with -1 marking pruned or out-of-bounds entries so they act as zero padding. Because ViT tokens sit on a known regular patch grid, the table is built in two linear passes: a dense lookup array from grid coordinates to sparse indices, then a fill step for each kept token's eight neighbors. Both run on CPU in under about 3 ms, negligible against the GPU-bound adapter pass, and the table is reused across adapter layers until the next selection. Custom CUDA kernels then gather neighbor features through the table; large channel counts use an implicit GEMM path via cuBLAS, smaller ones a fused tiled kernel.

Where sparse convolution overtakes dense convolution

The primitive is not faster everywhere. Kernel timing on THUMOS-14 shows sparse and dense convolution reaching parity at roughly a 55% keep rate; below that threshold sparse is strictly faster, reaching 8x speedup at a 10% keep rate. Dense convolution stays nearly flat across keep rates because it still computes over the full grid.

That crossover matters because SV-TAD prunes progressively. Token selection is applied at evenly spaced blocks (4, 8, 12, 16 for ViT-L), and the first three blocks keep standard 2D convolutions since sparsity offers nothing in the dense regime. With a per-stage keep rate of 0.6, the effective rate falls to 0.36 after two stages and 0.13 after four, so most layers sit well below the crossover point. Savings therefore compound across depth rather than coming from a single layer.

Accuracy holds while throughput scales

On VideoMAEv2-L, integrating the primitive into the SV-TAD adapter framework reduces computation by up to 64% and delivers 2.2x faster inference, while accuracy on THUMOS-14 and ActivityNet-1.3 stays at state of the art. Scaled to InternVideoNext-L, the approach surpasses the previous state of the art at roughly half its computational cost.

The keep-rate sweep on THUMOS-14 with InternVideoNext-L shows the trade-off is gradual rather than brittle. At a 0.7 keep rate the model reaches 87.05 TFLOPs, 0.42 videos/s, 78.22 mAP@0.5 and 74.13 average mAP. Dropping to 0.5 gives 60.13 TFLOPs and 0.60 videos/s, with mAP@0.5 at 76.40 and average mAP at 73.00, a loss of 1.13 average mAP points. The authors conclude the keep rate can be tuned per deployment budget; note that these throughput and accuracy figures come from one backbone and one dataset, not from every setting tested.

Auxiliary tokens ride the sparse path

Because the sparse formulation never rebuilds a dense grid, it can carry extra task tokens alongside the visual ones. SV-TAD adds learnable auxiliary embeddings that query the sparse bottleneck features through a lightweight cross-attention module, updating only the auxiliary tokens and leaving the visual path untouched. Importance scores for token selection can then incorporate attention from all auxiliary tokens, biasing which patches survive toward regions relevant to the auxiliary task; with only the class token present, this reduces to the standard formulation.

The paper demonstrates this with landmark-guided token selection and pose heatmap prediction on fine-grained assembly detection, reporting improved results on ATTACH. The same cross-attention also serves as the adapter's mechanism for modeling temporal dynamics, and the class token acts as a relay that frozen self-attention propagates to visual tokens at the next layer.

What the speedup costs in memory

Peak GPU memory stays comparable between SV-TAD and the dense AdaTAD baseline. Frozen backbone activations dominate the footprint, and the sparse kernels replace rather than augment the convolution pathway, adding only the neighbor index table, about 0.2 MB at a 0.6 keep rate. The reported throughput and FLOP reductions therefore come without a memory penalty in the configurations tested.

Two boundaries qualify the result. The sparse kernel only beats its dense counterpart below the roughly 55% keep rate, so the primitive pays off when progressive pruning through the layers is acceptable. And the CUDA implementation ties the current speedups to that backend, although the authors describe the core algorithm as architecture-agnostic and note that porting to other accelerators would require replacing it.

Choosing a keep rate for deployment

The practical decision this paper supports is not whether to prune but how far. Because accuracy degrades smoothly as the keep rate falls, the choice becomes a budget question: a 0.7 keep rate on InternVideoNext-L preserves the best accuracy-throughput trade-off in the reported sweep, while 0.5 buys 0.60 videos/s instead of 0.42 for 1.13 average mAP points. The 55% crossover sets the floor below which sparse convolution stops paying for itself, and the staged schedule is what pushes most layers past it.

The paper's distinctive claim is that token selection and convolutional adapters no longer have to trade off against each other. Earlier adapters saved parameters but not inference computation, and earlier pruning saved attention but forced dense reconstruction. SV-TAD's contribution is the primitive that removes that reconstruction step, which is why the reported savings appear as wall-clock speedups rather than only as FLOP counts.

Sources used in this answer