From the creator of Redis; run LLM locally with ds4

What it claims

ds4 runs DeepSeek V4 Flash at a 2-bit quantization (Q2) on a single high-memory machine. The page gives a reference figure for an M5 Max with 128 GB at 32K context: 34.4 tokens/s generation and 557 tokens/s prefill. The supported families include text and vision models. The page says each supported layout is checked against the official model's outputs; it is not described as merely loading.

Evidence

The only number on the page is the M5 Max 128 GB, 32K-context reference row above. It is quoted from the upstream ds4 benchmark table. There is no comparison against another engine or against an unquantized baseline. The page advises reading prefill and generation separately, especially for long-context agent workloads. At these figures, prefill runs roughly 16× faster than generation. Long prompts are therefore a smaller cost than long outputs, and the SSD prefix cache exists to avoid paying for prefill again.

How to use it

The fit check on the page gives a conservative path by memory size:

• Baseline: V4 Flash Q2.
• At 128 GB: GLM 5.3 Q2 and Qwen Q4 also fit.
• V4.1 Q2: streams weights from SSD rather than holding them fully in memory.
• 64 GB: the Qwen model is the one that fits ("Qwen on 64GB").

The starting command for the baseline is:

./download_model.sh ds4f-q2 && make

The flow is:

1. Fetch the project GGUF.
2. Build for your backend (Metal, CUDA or ROCm).
3. Run ./ds4 for chat, ./ds4-server for the local API, or ./ds4-agent for persistent coding sessions.

Because ds4-server speaks OpenAI- and Anthropic-style APIs, existing coding agents and editors can connect by changing only their base URL.

Limits

• Generic GGUF files are explicitly not supported. You must use the project's own GGUF builds, so you cannot bring your own quantization or fine-tune.
• Model support is described as "a small, opportunistic set". Anything outside DeepSeek V4/V4.1 Flash, GLM 5.x and Qwen3.8 Flash Next is unsupported.
• V4.1 Q2 depends on SSD streaming on 128 GB machines. The page gives no throughput figure for that mode, so expect it to be slower than the baseline and benchmark it yourself.
• The page publishes no CUDA or ROCm numbers and no quality or accuracy figures for the 2-bit builds. The fit check calls its guidance estimates and points to a separate Hardware and Installation guide for details.