Skip to content

Back to the catalog

New LoRA Skills Should Read but Never Write
arxiv.orgpaper

New LoRA Skills Should Read but Never Write

This paper introduces READ (Read-only Expansion of Adapter Deltas), a method for composing multiple independently trained LoRA adapters into a single model without the weight-space interference that plagues naive merging. READ rewrites each adapter into a balanced canonical factorization and adds a one-directional coupling so new skills can read old adapters' input subspaces but never write into their output subspaces, requiring only a small trainable coupling row per new skill and folding cleanly into base weights with no inference-time routing cost. Evaluated across four benchmark suites and two model families, READ outperforms the strongest published composition baselines by over 20 points on SuperGLUE and over 7 points on a domain suite when adding skills sequentially.

TL;DR READ composes multiple LoRA adapters one-directionally (new skills read old ones, never overwrite them), beating prior merging baselines by 20+ points on SuperGLUE.
Open original ↗
Document

What it claims

READ composes independently trained LoRA adapters into one folded weight update. Each new skill trains only a small coupling row. That row lets the new skill read old adapters' input subspaces, and old skills' outputs are left unchanged by construction. The authors tested it on Llama-3.2-3B and Qwen3-4B across GLUE, SuperGLUE, a Domain suite and BBH, adding one skill at a time.

Across 32 addition lineages, READ beats the strongest of fourteen foldable alternatives per lineage by +0.073 on average (95% CI +0.047 to +0.101). It wins 24 lineages and loses 8. All eight losses are on BBH.

How it works

The setting is a bank of k adapters with stacked factors and a coupling matrix G: ΔW = B_stack · G · A_stack. G is built from r×r blocks:

  • Diagonal block G_ii scales and mixes skill i's own directions.
  • Off-diagonal block G_ij maps skill j's input directions into skill i's output directions.
  • G = identity gives the naive sum of adapters.

The method turns on two choices.

  1. Canonicalize. A LoRA factorization is ambiguous, because (BR)(R⁻¹A) = BA for any invertible R. A coupling learned on raw factors would encode quirks of initialization, data order and seed. READ fixes this before any coupling is trained: - Take thin QR decompositions: B = Q_B R_B and Aᵀ = Q_A R_A. - Take the SVD R_B R_Aᵀ = UΣVᵀ. - Set B^c = Q_B U Σ^½ and A^c = Σ^½ Vᵀ Q_Aᵀ.

The update B^c A^c = BA is preserved exactly, and both factors share the metric Σ.

  1. Read-only coupling. The write column G_{≤k,k+1} is fixed at zero. Only the new row G_{k+1,≤k} and the new diagonal block G_{k+1,k+1} are trained: - The diagonal block is initialized to 0.3·I. The row blocks start at zero. - The old G block, all factors and W₀ stay frozen. - Training is one pass over the union of all task data D₁:ₖ₊₁.

Write blocks would receive gradients from old-task examples. Read blocks only affect the new skill's output.

  1. Fold. B_stack·G·A_stack is added into W₀. Serving runs the plain base model, with no router, task ID or per-task head. The task instruction in the prompt selects the label space.
flowchart LR
  Old[Frozen old bank] -->|read: trainable row| New[New skill output]
  New -.->|write: fixed to 0| Old
  Old --> Fold[Fold into W0]
  New --> Fold

Source adapters in the experiments are rank 8, on the q- and v-projections of every attention block. They are trained as generative classifiers: labels are emitted as text through the shared LM head.

Evidence

All numbers are terminal suite averages against the strongest foldable alternative built from the same adapters:

ModelSuiteREADStrongest alternative
Llama-3.2-3BSuperGLUE0.7830.605
Llama-3.2-3BDomain0.8870.846
Qwen3-4BGLUE0.8380.775
  • READ also wins GLUE on Llama and the GLUE, SuperGLUE and Domain suites on Qwen, but the page does not give those numbers.
  • The baselines discussed include TIES, DARE, LoRA-LEGO, KnOTS and RegMean-style merges. Routing methods such as LoRAHub, LoRA-Mixer and GraftLLM are positioned as non-foldable.

How to use it

Costs per append, as stated:

  • Canonicalization: O(dr² + r³) per module, under a million flops at Llama sizes.
  • Trainable parameters: (k+1)r² floats per module. At a six-skill append that is 384 per module, 21,504 in total.
  • Storage beyond the adapters: the coupling, (kr)² floats per module. That is 129k across the 56 Llama q/v projections.
  • Fold: O(d²kr) multiply-adds, done once.
  • Serving: after folding, inference costs the same as the base model.

The visible text names no code release, weights or licence.

Limits

  • Old task data required. Each append trains on the union of old and new task data. This is not data-free composition, even though the paper cites joint retraining cost as a motivation.
  • BBH regresses. All eight lineage losses are on BBH, so check that suite if reasoning-style tasks matter to you.
  • Coupling grows with the bank. Effective rank grows as kr, coupling storage as (kr)², and fold cost scales with k.
  • Narrow test range. Only 3–4B models, rank-8 q/v adapters and classification-style tasks with text labels were tested.
  • Routing still has a place. The authors say routing remains appropriate when each solo adapter must stay intact per task. Their §5 discussion of this boundary is not in the fetched text.
  • Truncated page. The fetched text stops in §3.5. Ablations, full tables and the stated limitations section were not available.

Organisations named here

  • SuperGLUE

Technologies named here

  • LoRA

Techniques named here

  • READ
  • LoRA

Related in training