colibri: pure-C engine for running huge MoE models by streaming experts from disk

What it claims

colibri v2.1.0 is an Apache 2.0 inference engine by Vincenzo Fornaro, written in C with one file per model family and no BLAS. It has 13 engines: 10 for language models, one for images (Qwen-Image-2.1) and two for decision models. Its design rule is that placement changes only speed. Router decisions and weight precision are the same whether an expert is answered from VRAM, RAM or disk. The headline example is GLM-5.2 (744B parameters), which uses about 40B per token. Only about 11 GB of those change from token to token. The README shows it starting with 9.9 GB resident.

How it works

Every layer runs five steps: route, union, place, overlap, learn.

• Placement. Attention, shared experts and embeddings stay in RAM. Routed experts sit on disk behind a per-layer LRU cache. A pinned hot set is learned from your own conversations and stored in .coli_usage, which is updated every turn.
• Reads. Each expert's three matrices are read in one pread. A loader pool fetches missing experts while resident ones compute.
• Prefetch. A router-lookahead thread prefetches the next layer. GLM-5.2's routing is 71.6% predictable one layer ahead.
• GPU. A GPU holds the hottest experts and the dense layers. Experts it misses are computed on the CPU in the same step.
• Long context. GLM-5.2's MLA keeps a compressed KV state of 576 floats per token instead of 32,768. It survives restarts, so a reopened conversation does not reread its prompt.
• Correctness. CI checks every engine token for token against a reference implementation (transformers) on a tiny fixture. It also checks Vulkan output against CPU tokens.

flowchart LR
  R[Router picks experts] --> V{Where is expert?}
  V -->|VRAM tier| G[GPU computes]
  V -->|RAM cache| C[CPU computes]
  V -->|miss| D[pread from NVMe] --> C
  G --> M[Merge layer output]
  C --> M
  M --> L[Update usage heat] --> R

Evidence

All figures are decode speeds unless noted.

GLM-5.2 int4 across hardware

Other models

• DeepSeek V4 Flash (284B): 0.93 tok/s on a Ryzen 7 5800X with 32 GB, CPU only. 1.5-1.6 tok/s on an RTX 5080 with two NVMe drives.
• Kimi K3 (2.8T): about 9.4 s per token, with experts read at 6.3 GB/s.

Machine D (i5-13500, 62 GB RAM, RTX PRO 4000 24 GB, two NVMe drives), measured through coli serve with a 450-token prompt and a 128-token answer:

• Qwen3.6-35B-A3B Q4_K_M: 104.8 tok/s on CUDA, 92.9 on Vulkan, 13.9 on CPU.
• Qwen3.8-Flash-Next IQ2_XS (68 GB, larger than the card): 51.4 tok/s on CUDA.
• At a 262,144-token context, Qwen3.8-Flash-Next decodes at 45.7 tok/s.

Integrated GPU (Radeon 780M)

• Vulkan helps Qwen3.6: 6.0 to 9.9 tok/s.
• Vulkan hurts OLMoE: 23.1 to 17.3 tok/s.

Other measured gains

• Two NVMe drives on independent controllers: +37.5% decode.
• Qwen3.6 MTP head (Q36_MTP=1) on CUDA: 99.7-104.5 to 115.8-120.1 tok/s.
• Parking a conversation between agent turns (COLI_PARK): Hermes Agent's 14,810-token second turn fell from 86 s to 4.9 s.

How to use it

The minimum is 8 GB of RAM and 22 GB of free disk for the smallest model. For streamed models, disk speed sets the decode rate.

git clone https://github.com/JustVugg/colibri
cd colibri
./start-here.sh

The setup detects your hardware, recommends a model and builds for CUDA, Vulkan or CPU. Useful flags:

• --list shows every model against your machine.
• --model ID installs a specific model.
• --backend chooses the build.
• --dir puts models on another disk.

Manual commands from c/:

./coli chat  --model /nvme/qwen36
./coli serve --model /nvme/qwen36
./coli plan  --model /nvme/qwen36
./coli tune  --model /nvme/qwen36
./coli import /nvme/Flint-UD-Q4_K_XL.gguf

APIs. coli serve exposes /v1/chat/completions (OpenAI-compatible) and /v1/messages (Anthropic-compatible, so Claude Code works). It also supports --kv-slots N, up to 16 concurrent conversations.

Decisions. POST /v1/systemone returns a probability for every allowed option instead of generated text. It is wire-compatible with TypeSafe's Jev API.

Multi-drive. COLI_MODEL_MIRROR reads from a copy of the model on a second drive.

GLM-5.2 weights. Use mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp (429 GB). Do not use the older per-row int4 mirrors: they score about 9 points worse on quality and caused the looping answers reported in issue #455.

Limits

• Large models are slow on modest hardware. GLM-5.2 cold on a 25 GB laptop runs at 0.05-0.1 tok/s.
• Integrated GPUs can lose to the CPU. Qwen3.6 on an Iris Xe ran at 1.7-1.9 tok/s with Vulkan versus 2.1 tok/s on CPU.
• Vulkan output can drift from CPU output by a word over long answers.
• The no-Resizable-BAR GPU path is untested on real such cards.
• DIRECT=1 (O_DIRECT reads) is neutral or worse on some drives.
• MTP speculation measured a 32% loss at around 85% expert hit rate.
• Learned expert pins can overfit a prompt.
• On WSL, keeping models under /mnt/c is many times slower.
• Tool calling is missing on Inkling and OLMoE.
• Qwen-Image-2.1 is licensed for non-commercial use only. Muse Spark Open is under Meta's licence agreement.