Ternary-Bonsai-2-27B: 27B model compressed to ~1.75 bits/weight, 98.2% of FP16 quality retained

What it claims

A Qwen3.8-27B reasoning model with every language weight ternary: embeddings, attention, MLP and LM head. It is meant to run interactively on a laptop or a single GPU. The page reports 84.78 averaged over 14 thinking-mode benchmarks. That is within 0.4 points of UD-Q4_K_XL at a third of its footprint, and more than 12 points above IQ2_XXS (72.59) at about 82% of its size. The model keeps the base model's 262K context, which stays practical because ~75% of the backbone's attention is linear. A vision tower ships separately as a Q8_0 mmproj file.

How it works

• Format: ternary g128. Each weight is {−1, 0, +1}, with one FP16 scale per 128 weights, which works out to ~1.71 bits/weight.
• Higher-precision tensors: 26.2M parameters stay above ternary. These are the linear-attention recurrent state path and the norms. Including them gives 1.72 bits/weight overall.
• Rotation: before ternarization, each matrix is rotated blockwise with a Hadamard transform. The rotation is folded into the stored weights offline, and the runtime applies the matching transform to activations. The file declares its rotation in metadata, so a runtime either applies it or refuses to load the file.
• Kernels: packed weights are consumed directly and never expanded to FP16.

Evidence

All results come from EvalScope + vLLM on H100, in thinking mode. They were measured with min_p=0.0, which differs from the recommended 0.05.

• Collapse shows up only on long-reasoning benchmarks. IQ2_XXS still scores 88.93 on MMLU-Redux but falls to 57.5 on AIME26 and 56.4 on LiveCodeBench. Bonsai scores 95.83 and 90.07 on those two. The page warns that this is why casual testing misses the degradation.
• By category: math goes from 97.06 to 96.57, coding is level with the baseline at 89.42, and agentic tool calling scores 74.92. The remaining gap is in knowledge/reasoning and vision.
• Intelligence density, defined as D = −log2(1 − score/100) / size_GB: Bonsai scores 0.457, against 0.257 for IQ2_XXS and 0.416 for the previous Bonsai 27B.
• Throughput (batch 1, llama-bench):
• ~130 tok/s on an RTX 5090.
• ~30 tok/s on a 72 W L4.
• ~28 tok/s on an M5 Pro, drawing 27.5 W on the GPU rail versus 300–455 W board power for the NVIDIA cards.
• ~47 tok/s on an M5 Max. This figure comes from the earlier pre-rotation build and is pending re-measurement.

How to use it

Runtime: you must use the PrismML-Eng llama.cpp fork. The MLX and mlx-swift forks cover Apple Silicon.

The hub's auto-generated snippets (stock llama serve … :F16, vLLM, Ollama) contradict the card. Stock llama.cpp behaves as follows:

• It rejects PQ2_0 and PTQ1_0 as unknown types.
• It loads Q2_0 silently and produces garbage, because it has no Hadamard runtime.

Choosing a packing:

flowchart TD
  A[Pick packing] --> B{Hardware}
  B -->|H100 / A100 / Blackwell / Apple| C[PQ2_0, 7.21 GB]
  B -->|Ada / L4 / tight memory| D[PTQ1_0, 5.95 GB]
  C --> E[Faster prefill everywhere]

PTQ1_0 moves 17% less data per decode step. Unpacking its dense trits costs extra arithmetic, though, so it loses on parts where batch-1 decode is bound by instruction throughput rather than memory bandwidth.

Run:

git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j    # drop -DGGML_CUDA=ON on macOS, Metal is default
hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .
./bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -c 32768 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 \
    -p "Explain quantum computing in simple terms." -n 16384

Generation settings:

• Reasoning effort defaults to xhigh, so leave a large -n. A cap such as 256 tokens cuts the output off mid-thought, before any answer appears.
• Use medium for shorter answers. low is unsupported and behaves close to xhigh.
• Instruct mode: temp 0.7, top_p 0.80, top_k 20, presence_penalty 1.5.
• For server use, tool calling, image input and speculative decoding, the card points to PrismML-Eng/Bonsai-demo. It calls that repo the source of truth and says it overrides the card where they disagree.

Limits

• Quality gap: the remaining loss is concentrated in knowledge/reasoning and vision.
• Footprint vs latency: PTQ1_0's smaller size does not translate into lower latency on Ampere, Hopper or Blackwell. The authors call fixing this ongoing work.
• Energy figures: Apple and NVIDIA energy are not comparable. powermetrics has no DRAM rail, while nvidia-smi includes HBM/GDDR.
• Measurement noise: the M5 Pro figure swings ~4% with background load.
• Licence: the page does not state one. Check the base model's terms and the repo before adopting.