Does freezing factor or spectral band matter more in low-rank LoRA?

Two design choices that spectral LoRA variants usually entangle

LoRA adapts a pretrained weight matrix W0 through a low-rank update sBA, where A maps the layer input into a small intermediate representation and B maps that representation to the output. Fixing one factor cuts the number of trainable parameters, but it also fixes either the input subspace or the output subspace available to the update. A spectral construction adds a second decision: which singular directions of the pretrained weight define that subspace.

Existing methods mix these decisions together. MiCA fixes the output factor on bottom singular directions, while PiCa's implementation fixes an input-side projector on top directions and shares the trainable output factor across layers. Comparing such methods changes orientation, band and sharing at once, so a score gap cannot be attributed to any single choice. This paper instead holds the model, data, adapted modules and training budget fixed and varies only two things: freeze A or freeze B, each on the top band (indices 1 to r of the thin SVD) or the bottom band (the last r directions).

The distinction matters because the two constraints are not symmetric, especially for rectangular layers. With B frozen, every output update lies in the column space of B. With A frozen, the update can act only through the input features represented by the rows of A. The paper's question is narrow and comparative: how large is the frozen-factor contrast relative to the top-versus-bottom spectral contrast, and how does that comparison change with rank?

At rank 2 the factor contrast exceeds the band contrast

The central low-rank result comes from a two-by-two design at rank 2, with learning rates selected separately for every configuration and test scores reported over five seeds. On Format / Llama-3.2-1B, A frozen reaches 77.4% on the top band and 78.2% on the bottom band in valid-format accuracy, while B frozen reaches 64.4% and 59.2%. On OpenBookQA / Qwen2.5-1.5B, A frozen reaches 53.1% and 52.5% multiple-choice accuracy, against 46.5% and 42.2% for B frozen.

The paper summarizes this with two contrasts. F is the factor contrast averaged across the two bands; S is the signed band contrast averaged across the two frozen orientations. At rank 2, the same-band advantage of freezing A is larger than either within-factor band difference on all four task-model pairs with complete comparisons. In other words, changing which factor is frozen moves the score more than switching between spectral extremes does.

The comparison is not a variance decomposition or a parameter-adjusted causal model, and the authors describe the factor-versus-band ordering as descriptive. The free-LoRA reference in the figure has rank 1 and a comparable, slightly larger trainable budget than rank-2 B-frozen LoRA, so its gap cannot isolate the causal effect of freezing a factor. Reference choice also matters: on Format / Llama-3.2-1B, equal-rank free LoRA scores 80.33% over five seeds with 2.15 times the B-frozen parameter count, which makes the top-band deficit 15.90 points rather than 12.41 against the rank-1 reference.

Freezing B trails free LoRA by 8 to 18 points

Beyond the internal two-by-two contrast, the paper compares B-frozen configurations against free LoRA at a nearby trainable-parameter count. Freezing B trails comparable-budget free LoRA by 8 to 18 percentage points on five task-model pairs spanning the formatting task and OpenBookQA. At frozen rank 2 the reference has rank 1, exceeding the B-frozen count by 7.5% on Llama-3.2-1B and 13.4% on Qwen2.5-1.5B; at frozen rank 16 the Qwen reference uses rank 7, within about 2.5% of the B-frozen count on the two Qwen sizes.

The authors call this a comparable-budget reference rather than an exact budget match, because it changes both the accessible update space and the rank. That is why the gap is reported as a descriptive deficit rather than as the isolated cost of freezing B.

The A-frozen advantage is not explained by parameter count alone. A-frozen adapters at equal rank contain 15.0% more parameters on Llama-3.2-1B and 26.8% more on Qwen2.5-1.5B, a consequence of the models' input and output widths. To test whether that count explains the result, the paper compares A frozen at rank 2 against B frozen at rank 3, giving B frozen the larger count on two models. The A-frozen advantage persists in a single-GPU-model replication, within individual MLP module groups, and when A is frozen on a random orthonormal basis on the two pairs tested.

The factor contrast weakens as rank increases

The complete A/B rank sweep covers ranks 2, 4, 8 and 16 on OpenBookQA with Qwen2.5-1.5B. Along that sweep the factor contrast decreases, becoming comparable to the band contrast at rank 16. The paper reports this trend descriptively rather than as a threshold, and it does not establish a universal rank at which the ordering flips.

Selected learning rates differ across configurations and ranks, which is part of why the comparison is framed as a controlled but not exhaustive search. For the complete rank extensions on OpenBookQA, the selected rates for B-top, B-bottom, A-top, A-bottom and the free reference are (5, 2, 5, 2, 5) x 10^-3 at rank 4 on Qwen2.5-1.5B, (5, 1, 2, 2, 2) x 10^-3 at rank 8 on that model, and (1, 0.5, 1, 0.5, 0.5) x 10^-3 at rank 16 on Qwen2.5-7B.

Scope limits how far the rank trend travels. The strongest evidence concerns Format and OpenBookQA. Larger models are tested on OpenBookQA, Mistral only at rank 2, and complete rank-4 and rank-8 comparisons cover one pair. ARC-Challenge shows different higher-rank behavior, and rank-32 results there concern B-frozen configurations only. The paper establishes neither a universal rank threshold nor a ranking of complete MiCA and PiCa methods.

A learned output subspace largely removes the B-frozen deficit

If the B-frozen deficit came from the rank constraint itself, no choice of fixed output basis should fix it. The paper tests this by extracting a subspace from a trained LoRA and using it as the frozen output basis. A trained oracle output subspace largely removes the low-rank deficit, which shows that the deficit depends on placement rather than on low rank alone.

The paper also probes cheaper, data-dependent alternatives. Neither random nor initial-gradient placement removes the observed deficit. On the formatting task, the initial-gradient placement gains 1.79 points over the top spectral band, with a nominal 95% interval of [-0.84, 4.42]. That interval excludes the large recovery seen with the oracle in this experiment, but the authors caution it should not be read as a statement about all gradient-derived subspaces, ranks or optimizers.

A warm-up diagnostic sits between these extremes. A free rank-2 LoRA trains for a chosen fraction of steps, then its output basis is frozen and A is retrained from zero over the remaining steps, with the total step count unchanged. The original direction seed and two new seeds give the same qualitative pattern: longer warm-up produces larger mean gains at the tested 5%, 10% and 25% cuts. At 10%, gains over top-band B frozen span 4.12 to 4.64 points on OpenBookQA and 6.09 to 7.79 points on the formatting task, and each of the four new comparisons passes the Bonferroni correction for that family. None of their corrected intervals establishes recovery of at least 70% of the earlier free-LoRA gap.

MiCA's PEFT implementation trails LoRA under a transferred recipe

The paper includes a follow-up on PEFT's MiCA implementation, which fixes the output factor on bottom singular directions. On OpenBookQA / Qwen2.5-1.5B at rank 16, that implementation trails comparable-budget LoRA by 3.08 points under a shared training recipe transferred from the MiCA paper. The transferred settings are explicit: init_lora_weights set to mica, r equal to alpha equal to 16, fixed B and initially zero A, learning rate 5 x 10^-4, four epochs, cosine schedule, 10% warm-up, weight decay 0.01, adapter dropout 0.05, gradient clipping 0.3 and batch size four without accumulation.

The authors are careful about what this comparison is. The original study uses a 512-token limit and normally adapts query and value projections, while this run retains all seven module types and the paper's own OpenBookQA loader, 384-token blocks, precision and evaluator. The base-to-instruct weight composition of the original benchmark is not part of this protocol, so the authors call it a transferred recipe rather than a full replication.

The shared-recipe test was planned after an earlier campaign result, in which MiCA minus the bottom B-frozen arm was 2.56 points with a [1.30, 3.82] interval, descriptively. The authors state that the shared-recipe comparison is an adaptively chosen follow-up, not an independent replication of MiCA, and that neither method is independently retuned in it. The follow-up covers one model, one task and one transferred recipe, without assessing acquisition-forgetting trade-offs.

An approximate multiplicity audit weakens several earlier claims

The paper's own uncertainty accounting narrows how firmly some results should be read. Five reporting seeds give limited precision for small effects and ratios, and intervals condition on the fixed test set. A seed-and-item bootstrap widened selected Format intervals by 1.8 to 2.6 times, and OpenBookQA lacks the retained predictions for that analysis.

Corrections apply within campaigns. An approximate 145-row Holm audit reduces historical positive decisions and excludes the new experiments. The factor-versus-band decomposition is post hoc, and the rank trend is reported descriptively. Retrospective power calculations often require 3 to 5-point band differences for 80% detection probability, so non-significance does not establish equivalence.

Hardware and selection add further qualification. Of 137 historical score cells, 86 use RTX 4000 Ada only and 51 mix GPU models, with six individual GPU records missing; the seven-arm Format replication uses RTX 4000 Ada throughout. Learning rates are selected on validation scores, normally over seeds 0 and 1, and some campaign grids end at a selected boundary, meaning an untested rate is not ruled out. A post hoc sensitivity analysis that chooses the best rate on reporting seeds is optimistic and is not a valid replacement for validation-based model selection.

Report orientation, basis and scale before comparing variants

The practical priority this paper supports is a reporting order, not a new default adapter. When comparing spectral LoRA variants, the frozen orientation, the basis construction and the scale should be stated before scores are compared, because at rank 2 the orientation choice moves results more than the top-versus-bottom band choice does. A comparison that changes orientation, band and sharing at once cannot attribute its gap to any one of them.

The second priority is the rank and budget regime. The factor contrast is largest at rank 2 and weakens along the OpenBookQA sweep, becoming comparable to the band contrast at rank 16, so a rank-2 finding should not be carried to higher ranks without rechecking. Equal-rank and comparable-budget references answer different questions: at equal rank, A-frozen adapters carry 15.0% more parameters on Llama-3.2-1B and 26.8% more on Qwen2.5-1.5B, while the rank-1 free-LoRA reference exceeds the B-frozen count by 7.5% and 13.4% respectively.

The third priority is the placement of the frozen subspace. Because a trained oracle output subspace largely removes the B-frozen deficit, and because random or initial-gradient placement does not, the deficit is best read as a property of where the fixed basis sits rather than as an unavoidable cost of freezing B. For readers deciding what to test next, the paper's own ordering is the useful one: fix orientation and basis reporting first, then check whether the rank-2 factor gap survives at the rank and budget actually in use.

Sources used in this answer