CoDyRA: Mastering the Plasticity-Stability Trade-off via Rank Minimization

TL;DR

The central challenge of Continual Learning (CL) is the "Plasticity-Stability" dilemma: how to learn new tasks without erasing old ones. CoDyRA (Continual Dynamic Rank-Selective LoRA) solves this by treating the effective rank of LoRA updates as its primary control knob. By minimizing update rank alongside training, CoDyRA implicitly regularizes forgetting, allowing models to absorb new knowledge directly into their backbones with minimal interference.

Background: The Invisible Sweet Spot

In the world of Parameter-Efficient Fine-Tuning (PEFT), LoRA is king. However, in Continual Learning, a fixed rank for all modules is a blunt instrument. The authors discovered through rigorous probing that:

1. Rank is a direct lever: High rank helps learning (Plasticity) but hurts memory (Stability).
2. Locations matter: The optimal rank for an Attention module is rarely the same as for an MLP module.
3. Tasks vary: No single rank is universally optimal; the "sweet spot" is dynamic.

Methodology: Rank Minimization as Stability

Instead of manually picking ranks, CoDyRA uses Capacity Control. It inserts LoRA updates into virtually all layers but attaches a learnable Importance Weight vector ($w$) to each rank component.

1. The Core Objective

The model optimizes a dual-purpose loss: ${\mathcal{L}}_{train}={\mathcal{L}}_{sup}+\lambda \sum ||w|{|}_1$ Here, ${\mathcal{L}}_{sup}$ drives the model to learn the new task, while the ${\ell }_1$ penalty on $w$ forces the model to use the fewest ranks possible.

2. Why Rank Controls Forgetting?

The authors provide a formal forgetting bound showing that forgetting ($F$) is bounded by the Frobenius norm and the effective rank of the update ($\Delta W$). Lowering the rank tightens this bound, ensuring the new task update doesn't "overwrite" the high-dimensional directions used by previous knowledge.

Experimental Battleground: CLIP and LLMs

CoDyRA was tested on diverse benchmarks including CLIP (Vision) and LLaMA/Gemma (Language).

SOTA Comparison

On the MTIL (Multi-Task Incremental Learning) benchmark, CoDyRA reached a Last Accuracy of 78.0%, outperforming standard vanilla LoRA and data-heavy reference methods. Remarkably, it was the only method where the model actually improved its zero-shot capabilities on unseen tasks (Transfer Accuracy: 70.1%) by merging knowledge effectively into the backbone.

Sublinear Parameter Drift

One of the most striking results is the Parameter Shift analysis. As the model learns task after task, the cumulative change in weights normally grows linearly, eventually leading to a "drifted" and broken model. CoDyRA’s cumulative shift grows sublinearly, being 6.7x smaller than fixed-rank LoRA after 10 tasks.

Figure: The model learns to allocate rank where it's needed—shifting from Attention in some tasks to MLP in others.

Deep Insight: "Take Only What You Need"

The philosophy of CoDyRA is "simplicity is stability." By pruning irrelevant rank dimensions to zero, the model creates "zero-overlap" subspaces for previous tasks. It doesn't need to store old data because it ensures that new updates are so compact they rarely touch the crucial weights formed by earlier tasks.

Limitations & Future Work

While CoDyRA is highly efficient (no extra inference overhead), it is "direction-agnostic"—it knows how much capacity to use but doesn't explicitly calculate the angle relative to old task gradients (Alignment-aware control). The authors suggest that combining rank minimization with explicit orthogonal constraints could be the next frontier in memory-free continual learning.

Conclusion

CoDyRA proves that rank minimization is more than just a compression trick—it's a fundamental regularizer for model stability. By adaptively selecting ranks, we can finally stop treating Large Models as fragile monuments and start treating them as evolving backbones that can learn indefinitely.