Sub-1-Bit LLM Compression via Latent Factorization

What it claims

LittleBit (NeurIPS 2025) compresses LLMs to between 1.0 and 0.1 bits per weight. It replaces each dense weight matrix with low-rank latent factors, binarizes those factors, and recovers magnitude through small learned scales. The model architecture stays the same at inference time. LittleBit-2 (ICML 2026) changes only the initialization: it aligns the SVD-derived factors with the binary hypercube before quantization-aware training (QAT). The authors state this adds no inference overhead. This repo is the official implementation of both papers.

How it works

1. Factorize each weight matrix into low-rank latent factors using SVD.
2. Optionally (LittleBit-2) apply Internal Latent Rotation with Joint Iterative Quantization (Joint-ITQ). This fixes what the authors call "latent geometry misalignment" between the factors and the binary values they will be quantized to.
3. Binarize the factors. Learned scales restore magnitude.
4. Run QAT with the SmoothSign quantization function and, optionally, residual factorization (--residual True).

flowchart LR
  W[Dense weight] --> S[SVD latent factors]
  S -->|default| Q[QAT: SmoothSign, binarized factors + scales]
  S -->|--use_itq True| R[Joint-ITQ rotation] --> Q
  Q --> D[Same deployed factorized layer]

Both initialization paths end in the same deployed layer. That is why LittleBit-2 costs nothing extra at inference.

How to use it

Supported model families: OPT, Llama / Llama 2 / Llama 3, Phi-4, Qwen2.5, QwQ, Gemma 2, Gemma 3 and Qwen3. The authors recommend Python 3.12, CUDA 12.4 and PyTorch 2.8.0. To reproduce the paper results, pin transformers==4.51.*.

conda create -n littlebit python=3.12
conda activate littlebit
conda install nvidia/label/cuda-12.4.1::cuda-toolkit -c nvidia/label/cuda-12.4.1
pip install torch==2.8.0+cu124 torchvision==0.23.0+cu124 torchaudio==2.8.0+cu124 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
pip install "transformers==4.51.*"

Single-GPU QAT example (Llama-2-7B, C4 + WikiText data). Add --use_itq True to get LittleBit-2:

CUDA_VISIBLE_DEVICES=0 python -m main \
    --model_id meta-llama/Llama-2-7b-hf \
    --dataset c4_wiki \
    --save_dir ./outputs/Llama-2-7b-LittleBit-2 \
    --num_train_epochs 5.0 \
    --per_device_train_batch_size 4 \
    --lr 4e-05 \
    --warmup_ratio 0.02 \
    --report wandb \
    --quant_func SmoothSign \
    --quant_mod LittleBitLinear \
    --residual True \
    --eff_bit 1.0 \
    --kv_factor 1.0 \
    --min_split_dim 8 \
    --l2l_loss_scale 10.0

--eff_bit sets the target bits per weight. For multi-GPU training, use deepspeed --num_gpus=4 main.py ... --ds_config_path configs/zero3.json.

Evaluation covers perplexity on WikiText-2 and C4, plus zero-shot BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c and OpenBookQA:

CUDA_VISIBLE_DEVICES=0 python eval.py \
    --model_id ./outputs/Llama-2-7b-LittleBit-2 \
    --seqlen 2048 \
    --ppl_task wikitext2,c4 \
    --zeroshot_task boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,openbookqa

Quantization settings are resolved in this order: explicit CLI arguments, then littlebit_config.json, then config.json. Older checkpoints that lack littlebit_config.json need --quant_func, --quant_mod and --split_dim passed explicitly.

Limits

• Licence: CC BY-NC 4.0, so commercial use is not permitted.
• No results in the repo: the README reports no accuracy or perplexity numbers and no memory or speed figures. Check the papers before judging quality at 0.1 bpw.
• Training required: this is QAT, not post-training quantization. The 7B example runs 5 epochs, and the README gives no GPU-hour estimate.
• Version sensitivity: the authors warn that newer transformers releases may change model internals or evaluation behaviour.
• No published checkpoints: the Hub model ID in the eval example (username/littlebit-llama-7b-0.1bpw) is a placeholder, not a real release.
• No deployment speed claims: the README does not say whether inference uses dedicated binary kernels or gets faster in practice.