ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning

What it claims

ProLand is an optimizer for LoRA that constrains the basis factor B to be orthonormal (BᵀB = I) without a retraction at every step. Its key property is that the optimization step changes BᵀB only at second order in the step size, so a separate Landing correction can restore orthogonality on its own schedule. The authors prove that, in exact arithmetic, the deterministic method reaches O(T^-1/2) product-space stationarity and O(T^-1) constraint violation. With stochastic gradients, the expected bounds are O(T^-1/4) and O(T^-3/2), using horizon-dependent momentum and step size. The constraint-violation bounds also hold along each trajectory after an initial transient.

How it works

The orthonormality constraint restricts only the factorization. The set of representable rank-at-most-r increments ΔW = BA is unchanged. Each iteration of Algorithm 1 runs these steps:

1. Project the stochastic gradient and the previous momentum onto the current first-order product space, using the projector Π = P_B Z + (I − P_B) Z P_A. This reprojection is needed because old momentum need not lie in the new space.
2. Form a Nesterov-style input J, take its polar factor Y = UVᵀ (the Muon step), then project Y back into the product space.
3. Realize that direction as D_A = B†Y and D_B = (I − P_B) Y A†. This choice guarantees BᵀD_B = 0 even when B is off the manifold.
4. Update A ← A − γD_A and B ← B − γD_B − ρB(BᵀB − I) at the same time. The last term is the Landing correction, a gradient step on ¼‖BᵀB − I‖²_F.

Because BᵀD_B = 0, the optimization direction and the Landing correction are Frobenius-orthogonal. Their step sizes γ and ρ can therefore be tuned independently.

flowchart LR
  G[Stochastic gradient] --> P[Project to product space]
  M[Previous momentum] --> P
  P --> Pol[Polar factor, reproject]
  Pol --> DA[D_A update to A]
  Pol --> DB[D_B with BᵀD_B = 0]
  C[Landing: -ρB·(BᵀB−I)] --> UB[B update]
  DB --> UB

The method has an exact low-rank implementation. Momentum factors have width at most 2r, and the polar factor is computed on a core of at most 2r×2r. It uses only the factor gradients BᵀG and GAᵀ, never the dense G. Excluding gradient evaluation, each step costs O((m+n)r² + r³) operations and O((m+n)r) extra memory.

Evidence

The method is evaluated on 4 GLUE validation tasks with DeBERTa-v3-base and on 7 language benchmarks with Qwen2.5-3B, Llama-3.2-3B and Moonlight-16B-A3B, at multiple ranks. The page states that ProLand has the highest task-averaged score in every model and rank configuration tested. Matched ablations show that the Landing correction substantially reduces constraint violation.

In a toy example (Figure 1), swapping Landing for per-step retraction gives a different stationary point. ProLand reaches an objective of about 0.5 versus 1.125 for the retraction variants, with a peak violation |BᵀB − 1| of 0.494 along the way. The named baselines in related work are Stiefel-LoRA (retraction), Manifold-LoRA (Landing with projected gradients), LoRA-Pro, sMuon and LoRA-Muon.

Limits

The fetched text contains no per-task scores, margins or compute and time comparisons. Only the "highest average" claim is visible. It also gives no code, repository or hyperparameter values.

The theory has several stated conditions:

• It assumes exact arithmetic. Finite-precision diagnostics are deferred to Appendix J.
• It requires A to stay well-conditioned (σ_r(A) ≥ a₋ > 0). This holds only after an initial phase, because LoRA zero-initializes a factor.
• It requires the initial violation to satisfy ‖C₀‖ ≤ ε < 1, with ρ ≤ 1/2 and the step size bounded in terms of a₋ and ε.

The stochastic rates also need the momentum and step size tied to the training horizon T.