TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

What it claims

TACO is a first-order optimizer for full-parameter LLM fine-tuning. Its update for each 2-D weight matrix is ternary and has exactly one nonzero per column. That sparsity comes from the closed-form steepest-descent solution under a dimension-normalized 1→1 operator norm, not from post-hoc Top-k sparsification. The authors pitch it as a Muon-style geometric optimizer that behaves like Adam rather than Muon when fine-tuning. This matters because the Muon/AdamW "optimizer mismatch" can degrade AdamW-pretrained models.

How it works

• Vanilla update. For an m×n weight with gradient G, pick each column's largest-magnitude entry, keep only its sign, and scale by m/n: D = (m/n)·S(G), then W ← W − η·D.
• The m/n factor comes from the dimension normalization and is not a tunable per-layer hyperparameter.
• The 1→1 norm gives each column its own ℓ1 budget, which is why the problem separates by column.
• Practical version. Minibatch noise makes the selected coordinate unstable from step to step. A dense gradient EMA would fix this but would also cancel the memory saving. TACO instead keeps a small set of dynamic FP8 "heavy hitter" gradient components per column and processes gradients immediately. This cuts persistent state from O(mn) to O(n).
• Hyperparameters. The paper ablates the history EMA decay, the heavy-hitter threshold, update sparsity, and the step budget. These are the knobs you would tune.

Evidence

• Convergence. Vanilla TACO reaches an ε-stationary point in O(ε⁻²) iterations under L-smoothness, measured in TACO's own dual norm.
• Adam-like behaviour (theory). The setting is simplified: a fixed input z, an objective that depends only on Wz, and a common target output. In it, TACO and an Adam proxy follow the same continuation set and converge to the same W*. A Muon proxy converges to a different solution, and the paper gives the gap in closed form.
• µP. If row "winner" counts are balanced (max cᵢ = Θ(n/m)), then ‖D‖₂ = Θ(√(m/n)), the standard µP spectral scaling. This suggests learning rates may transfer across widths, but it rests on that assumption.
• OPT-13B on SST-2 (single H100 80 GB).
• TACO: 94.22% accuracy at 27.54 GB peak memory, the low-memory end of the Pareto frontier.
• Full Adam: 95.3% at 254.8 GB.
• So "comparable accuracy" here means about 1.1 points below full Adam.
• The figure also compares against AdamW8bit, FlashAdamW, GaLore, APOLLO-Mini, Adafactor, Muon, LoRA+Adam, MeZO, ZO-Muon and SubZero+LoRA, but the extracted text does not give their individual numbers.
• OPT-30B on one H100.
• Peak memory per task: SST-2 62.5, RTE 64.7, BoolQ 68.1, COPA 62.5, CB 65.8, MultiRC 70.4, SQuAD 64.6, DROP 68.2 GB.
• Optimizer state is 0.267 GB, under 0.5% of peak memory. The variation across tasks comes from transient memory.
• Adam does not fit this model on the card.
• Coverage. Full fine-tuning of Qwen3-32B on one H100 is also reported. Experiments span the OPT, Qwen3, Pythia, Llama and Mistral families.

How to use it

• Code. Listed as github.com/Jichao2357/TACO optimizer. The space suggests a separator was lost in text extraction, so verify the exact repo name.
• Install and licence. The fetched text gives no install command, API or licence.
• Scope. The update is defined for 2-D weight matrices. The fetched text does not say how 1-D parameters such as norms and biases are handled.
• Hardware. For 30–32B models, plan on about 62–71 GB peak on an 80 GB GPU. Activations and transient buffers, not optimizer state, are what limit you.

Limits

• Truncated page. The text cuts off in Section 4. Section 5 (implementation details) and Section 6 (results tables and ablations) were not available, so per-task accuracy beyond OPT-13B SST-2, runtime numbers and ablation outcomes are unverified here.
• Proof scope. The theorems cover vanilla, history-free TACO. The Adam-equivalence and µP results hold only under the stated simplified assumptions, not for the practical FP8 heavy-hitter variant.
• Evidence base. The tasks are classic fine-tuning benchmarks (SuperGLUE-style, SQuAD, DROP). Instruction tuning and long-context workloads are not shown in the available text.
• Status. This is a v1 arXiv preprint dated 1 Oct 2026.