Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

What it claims

ZFO wraps a standard first-order optimizer such as AdamW or Muon. The optimizer's update vector g_t fixes the direction. Two extra forward passes on the same minibatch then pick the step length. The paper offers four step-selection variants: second- and third-order Taylor (T2, T3) and second- and third-order Padé (P2, P3). The authors say no single variant wins everywhere: "the magnitude and preferred local model depend on the objective." They also claim ZFO is robust to the initial learning rate. The paper is listed as NeurIPS 2026.

How it works

Each step does the following:

1. Get the direction. The base optimizer produces g_t. Normalise it to z_t = g_t/‖g_t‖.
2. Restrict the search. Look for a step s only in I_t = [0, R_t], where R_t = β·α·‖g_t‖. The usual optimizer step is α‖g_t‖. With β = 1 the interval ends exactly there, so ZFO can only shorten the step. You need β > 1 to allow longer steps. The learning rate α still matters because it sets the search radius.
3. Collect four quantities. - From the normal forward pass: ϕ(0). - From the backward pass: the directional derivative a = ⟨∇F, z⟩. - From two extra probes: ϕ(±ε), evaluated on the same prompt-response pairs.
4. Estimate curvature. Compute b = (ϕ(ε) − 2ϕ(0) + ϕ(−ε))/ε². Because the gradient is reused, the same two probes also give c = 3/ε³·(ϕ(ε) − ϕ(−ε) − 2εa).
5. Fit and maximise a local model over I_t: - T2: check the endpoints and the stationary point −a/b. - T3: check the endpoints and the real roots of a + bs + ½cs². - P2 and P3: rational models with q₂ = −b/(2a) and q₃ = −c/(3b).
6. Fall back if needed. If a Padé denominator has a pole inside I_t, or its coefficients are ill-defined, ZFO uses the T2 step instead.
7. Update θ ← θ + s·z.

flowchart LR
  A[Base optimizer direction] --> D[Local model fit]
  B["ϕ(0), ϕ'(0) from fwd/bwd"] --> D
  C["2 probes ϕ(±ε)"] --> D
  D --> E{Padé pole in interval?}
  E -- no --> F[Model-optimal step]
  E -- yes --> G[Fall back to T2 step]

Relation to GeN, the generalized Newton step-selection method: the authors treat GeN as one quadratic instance of ZFO. The differences are that GeN rejects steps based on curvature, while ZFO bounds the search interval, reuses the backward-pass derivative, and so gets third-order information from the same two probes.

The objectives are written as maximisation (gradient ascent). The minibatch S_t can hold supervised labels or sampled responses, so the method is not limited to supervised fine-tuning.

Cost: two extra forward passes per step, compared with the many evaluations a full line search would need.

Evidence

Theorem 1 (curvature estimates). With probability at least 1 − δ, both the b and c estimates are within a finite-difference bias plus σ·√(2 log(4/δ)/n) of their population values. The bias is Bε²/12 for b and Lε²/20 for c.

Theorem 2 (step quality). The T2-selected step is near-optimal over [0, R] for the fixed-batch objective.

Convergence. ZFO reaches a neighbourhood of a stationary point. The size of that neighbourhood is set by the accumulated step-selection error.

Experiments. The fetched text stops before the experiments. It names only "controlled low-dimensional problems and LLM fine-tuning tasks." It gives no models, datasets, metrics or numbers.

How to use it

Code: https://github.com/nizswan/Zeroth-First-Order-Framework

The page gives no install or run instructions. Hyperparameters you add on top of the base optimizer:

• ε: probe size.
• β: search radius multiplier.
• Model choice: T2, T3, P2 or P3.

Limits

• No empirical results were captured. Before adopting, check the paper's tables to see which variant won on which objective and by how much.
• Theorem 1 holds only for a fixed direction. The authors state that in Algorithm 1 the direction is computed from the same batch as the probes. The independence the concentration argument needs therefore does not strictly hold in practice.
• Strong smoothness assumptions. The analysis assumes each per-sample directional objective is C⁵ with bounded derivatives.
• Convergence is only to a neighbourhood of a stationary point, not to a stationary point.
• Default β = 1 cannot exceed the baseline step length.