MakazhanAlpamys/Soup

What it claims

Soup (soup-cli, Apache-2.0) fine-tunes and post-trains any AutoModelForCausalLM HuggingFace model from one YAML + one command, wrapping transformers/PEFT/TRL/MLX with auto batch-size, GPU detection, and QLoRA. The headline is layer streaming: Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512 at 119.6 tok/s, 3.32 GB peak on an RTX 3050 Laptop 4 GB, reproduced on an H100 at 113.00 tok/s in the same 3.32 GB. Both figures were measured on v0.72.2, before a v0.73.0 correctness repair that cost −4.8% at 32B; neither has been re-run on a 4 GB card (pending, issue #361). Layer streaming is opt-in (stream_layers: true) and still BETA.

How to use it

pipx install soup-cli            # light CLI: config + data tools, no PyTorch
pipx install "soup-cli[train]"   # add training stack (torch, transformers, peft, trl)
pipx install "soup-cli[all]"

soup init --template chat        # templates: chat, code, tool-calling, dpo/orpo/simpo/ipo/kto, moe, vision, audio…
soup train --config soup.yaml

Use double quotes on the extras ("soup-cli[train]") — single quotes fail in cmd/PowerShell. Constraints the summary omits: Python 3.10–3.12 only (3.13+ unsupported, untested PyTorch wheels crash the native extension); requires torch>=2.6.0 as of v0.75 (2.5.1 broke every preference trainer via trl>=0.29); 8 GB+ VRAM for 7B QLoRA under normal (non-streaming) runs.

v0.75.0 breaking changes worth knowing before adopting

• Unknown config keys now fail (exit 1 / ValueError) instead of being silently dropped. A typo like quantizaton used to run with the setting simply not applied. Fix or remove such keys.
• MLX silently ignored six validated options before this release — train_on_responses_only, warmup_ratio/scheduler/weight_decay/optimizer, max_grad_norm, gradient_accumulation_steps, gradient_checkpointing. Only 8 of 32 optimizer names have an MLX equivalent; the other 24 are now refused by name. If you ran MLX on <0.75, your config may not have trained what you specified.
• grpo_variant: gspo now implements the published sequence-level objective (arXiv:2507.18071), replacing a column-centering heuristic where a padding token shifted every row's gradient. Existing gspo configs will not reproduce prior runs.
• Validation loss was computed on every backend and discarded until now.

The v3 preprint retraction

The Zenodo preprint (concept DOI 10.5281/zenodo.21771064; v3 = 10.5281/zenodo.21918325, current 13 Aug 2026) withdraws its own central explanation: the claim that layer streaming is "bound by host-to-device transfer, not the GPU" was an unmeasured inference from the H100 replication and is false at the published config. Measured: deleting every host-to-device byte buys only 1.4%, the compute stream waits on a copy 0.20% of the step, and the largest streaming-specific cost is per-layer NF4 dequantisation at 9.8%. The 8B-on-4GB claim and every measured number are unchanged. Also documented: a silent wrong-gradient defect on NF4 above ~165 MiB/layer where the forward stayed bit-exact and loss looked healthy while gradients were wrong (cause in the upstream library, now repaired and gated on real 32B/72B); bit-exactness verified forward 0.5B–72B, backward at 8B/14B; and a DeepSpeed comparison where eight cards of ZeRO-3 are slower than one card training resident.

Limits

Layer streaming's flagship 4 GB numbers predate the correctness repair and have not been re-measured on that hardware. Streaming is BETA and opt-in. Benchmarks (including failures and discarded numbers) live in benchmarks/; a free Colab T4 check caps the process to 4 GB and asserts bit-identity. Built and maintained on a single 4 GB laptop, so multi-GPU, 8B+ validation, and Apple Silicon work is hardware-gated rather than claimed.