Why do hybrid language models underuse their recurrent memory pathway?

Attention recalls details while recurrence aggregates context

Hybrid language models interleave attention layers with recurrent layers, giving the model two routes for carrying past information into the current input. The paper frames these as complementary: attention layers support precise memory recall by directly accessing earlier tokens, while recurrent layers synthesize information distributed across long contexts.

The authors extend that framing with their own observation: attention matters more when the answer appears directly in the context or can be found from a single piece of evidence, whereas the recurrent state matters more when the answer must be inferred rather than copied, or requires combining multiple pieces of evidence across the context. This distinction sets up the paper's central question of how well hybrid LMs actually use both pathways.

Masking one pathway reveals which one carries the answer

To separate the two routes, the authors split each input into a past portion and a current portion. In question answering, the past portion is the context and the current portion is the question. The unmodified full condition lets both attention and recurrent layers move information from past to current.

Two interventions isolate each mechanism. In the recurrent-only condition, attention is blocked across segment boundaries, so past information reaches the current portion only through the recurrent state. In the attention-only condition, the standard causal attention over preceding context is kept, but the recurrent state is prevented from propagating across segment boundaries, so preceding-segment information stays reachable only through attention. All layers remain active within each portion in both settings.

The paper also defines attention-only retention as the fraction of full-model accuracy preserved when cross-context information is accessible only through attention layers. Higher retention means greater reliance on attention and less on the recurrent state, which is the metric readers should track rather than raw accuracy alone.

Two hybrid models trained on QA and agentic tasks

The experiments cover two recurrent-attention hybrid LMs: Qwen3.5-4B, with 24 Gated DeltaNet and 8 full-attention layers, and Nemotron-H-4B-Instruct, with 24 Mamba2 and 4 full-attention layers. All models are trained with LoRA on all layers at rank 16, scaling factor 32, and dropout 0.05, using AdamW at a learning rate of 1e-4, a cosine schedule, a warmup covering 3% of steps, and an effective batch size of 16.

Because objectives with an auxiliary loss need extra computation, the authors halve the number of optimization steps to match total training compute with standard SFT, and use the same optimization setting across compared variants unless stated otherwise.

Evaluation spans long-context question answering and long-horizon agentic tasks. Question answering pools training examples from DROP, HotpotQA, Qasper, and NarrativeQA, yielding 12k training and 2k validation instances, with evaluation on each dataset plus MINTEval and MuSR, for 8.2k test examples at an average context length of 84k tokens. Agentic tasks use TextWorld and BabyAI, where agents must accumulate information across sequences of observations and actions.

Standard fine-tuning deepens the attention imbalance

The paper's key observation is that standard SFT increases hybrid LMs' use of the attention pathway while yielding little improvement in their use of the recurrent pathway. For Qwen3.5-4B on question answering, full accuracy rises from 30.2% at base to 34.6% after standard SFT, and attention-only accuracy rises from 27.0% to 78.5%. Recurrent-only accuracy moves only from 20.2% to 25.0%.

On agentic tasks the same pattern appears in success rate: full performance goes from 5.1% to 5.0%, attention-only from 2.5% to 4.0%, and recurrent-only stays at 5.0%. The model retains most of its accuracy when the recurrent pathway's access to past information is limited, but accuracy drops substantially when the same restriction is applied to attention.

Attention-only retention confirms the shift across question types. After SFT, retention is higher for verbatim than non-verbatim questions (63% vs. 46%) and for single-evidence than multi-evidence questions (60% vs. 50%). The authors read this as consistent with attention layers recovering precise details while contributing less when information must be abstracted or integrated.

An auxiliary loss that masks attention raises recurrent use

To encourage coordination, the authors add an auxiliary loss that uses the same supervised next-token prediction objective but masks attention to the preceding context. Recurrent layers stay unrestricted and process the entire sequence, so during the auxiliary forward pass information from the preceding context can influence the answer only through recurrent memory. The two terms are optimized jointly as L = L_SFT + lambda * L_rec, where lambda controls the auxiliary supervision strength.

On TextWorld Quest, adding the auxiliary loss improves over standard SFT (lambda = 0, 78.5% success rate) for all values of lambda, and the authors use lambda = 0.25, which achieves the best performance, in all experiments unless otherwise specified.

Across six QA datasets, the auxiliary objective raises Qwen3.5's micro-averaged accuracy from 34.6% under standard SFT to 38.5%, and Nemotron-H's from 31.0% to 36.2%. The gains are not uniform: Qwen3.5 improves on Qasper (30.1% to 35.9%), NarrativeQA (24.4% to 33.7%), and MINTEval (21.1% to 28.9%), while DROP slips slightly from 48.6% to 46.9%. Ablations show that recurrent-only supervision (L_rec) reaches 30.4% and attention-only supervision (L_attn) 31.6%, both below the combined objective, and adding L_attn to SFT reaches 35.6%.

Auxiliary supervision keeps improving as attempts accumulate

In the agentic setting, the authors test whether agents learn from their own failures by retrying a failed game up to a maximum number of attempts, with previous trajectories provided in context. The auxiliary loss improves the cumulative recovery rate by 19.2% on average over TextWorld.

The comparison matters more than the average: the standard SFT model tends to converge as the number of attempts increases, whereas the model trained with the auxiliary loss continues to improve. The authors interpret this as better learning from past failure trajectories, which is a claim about the training signal rather than a direct measurement of internal memory use.

The same imbalance appears in textual and latent memory

The authors extend the analysis to attention-based models without built-in multiple memory pathways, giving Qwen3-4B-Instruct two common forms of memory: textual memory, where a document is summarized and the model answers conditioned on the summary and question, and latent memory, where each document is represented as a layer-wise KV prefix with K memory slots trained in two stages.

Under matched memory budgets, textual memory performs better on verbatim and single-evidence questions, while latent memory is more competitive on non-verbatim and multi-evidence questions, mirroring the attention-versus-recurrence split. With a textual memory budget of 8k tokens and a latent memory budget of 128 slots, an oracle that selects the correct prediction whenever either single-memory model succeeds outperforms the single-memory models by 10.1% on average, indicating complementary information. Yet the model trained with both memories under standard SFT remains 7.9% below the oracle and even falls below textual memory alone (24.1% vs. 25.9%).

Adding an auxiliary loss that assigns each memory an explicit role changes this. The VERBATIM criterion, which assigns verbatim examples to the text role and non-verbatim examples to the latent role, achieves the best accuracy, improving it from 24.1% to 28.6%, and the largest increase in latent-memory use (from -2.4 to +4.7) while largely preserving textual-memory use (+11.4 to +10.4). OUTCOME reaches 26.1% with a smaller latent-use gain, REVERSE increases latent use only modestly while reducing textual use, and RANDOM does not improve accuracy. Supervision without per-example role assignment helps less: LATENT-AUX reaches 26.1% but at the cost of textual-memory use, TEXT-AUX mainly increases textual-memory use with little accuracy gain, and UNROUTED improves neither.

Role assignment, not just extra supervision, drives the gain

The comparison across role-assignment criteria is the paper's most decision-relevant result. Increasing the use of a single memory can help, but the largest gains come from assigning each example to the memory suited to it, according to the complementary strengths of the two memories. The VERBATIM criterion, which routes by whether the answer appears word-for-word in the context, outperforms random assignment, the reverse assignment, and per-example correctness-based assignment.

This is also where the paper's scope matters. The imbalance and the benefit of the auxiliary loss are reported across multiple recurrent-attention LMs in question answering and agentic tasks, and extend to attention-based LMs combining different forms of memory. The evidence is bounded to these models, tasks, and training configurations, including the LoRA setup and the compute-matched halving of optimization steps, so the specific accuracy numbers should not be read as general benchmarks for other architectures or training regimes.

For readers deciding what to take away, the priority is the routing rule rather than the auxiliary loss alone: the paper's strongest gains come from matching each example to the memory pathway whose strengths fit it, with verbatim versus non-verbatim as the criterion that worked best in these experiments.

Sources used in this answer