peonist-ai/halogen-flash-server

What it claims

halogen-flash-server serves Qwen3.8-Flash-Next only on the AMD Ryzen AI Max+ 395 (Strix Halo). Every kernel targets that one GPU and model family, with "no portability layer, no fallback path." The default checkpoint (v2) is 179.55B parameters at 4.16 bpw across trunk, experts and draft head. Counting the FP8 n-gram lookup table, it is 5.25 bpw. docs/QUANT.md breaks precision down by tensor family. The current image is 0.18.22, which adds a user-supplied d1-2B model on the NPU for one-pass decisions over text, images and video with bounding boxes. Flash replies are byte-identical to 0.18.21.

How it works

Speculative decoding uses two draft sources since 0.6.0: the model's own draft head and prompt lookup from the request's text. At temperature 0, output is byte-identical to serial greedy decode, and the page says this is verified on every release. Since 0.7.0 the engine opens a llama.cpp GGUF of the same model directly, such as unsloth's UD-IQ4_XS, with no conversion step. The same speculation and identity guarantee apply.

Evidence

All figures use the 0.18.0 image at defaults (v2 checkpoint, prompt cache on, greedy, thinking off) on a 128 GB Ryzen AI Max+ 395 at about 85 W sustained package power.

• Decode is 94 tok/s on repetitive output ("print the word red ten times" and four similar prompts), 73 writing code, 54 prose, 81 editing code, 71 for an agent tool call and 52 in six other languages.
• Prefill on a synthetic prompt is 1,821, 1,740 and 1,675 tok/s at 8,192, 32,768 and 131,072 tokens. Real text through the chat template runs about 8 to 12% slower.
• An agent turn at 30,000 to 96,000 tokens of context takes 4.2 s end to end, with the first token at 1.6 s. This is the median of 115 turns across three coding-agent sessions fixing a Python bug, timed on the client, with about 200 answer tokens each.
• A cold first turn of a 34,000-token prompt takes 21 s.

Prefill rows and a ten-shape decode mean reproduce with one command from the image. The page reports no comparison against llama.cpp or other runtimes.

How to use it

The BIOS UMA frame buffer must be set to its minimum, not Auto. The engine wants host RAM rather than dedicated VRAM.

mkdir -p ~/halogen-models
podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.18.22

Weights come from peonist-ai/halogen-qwen3.8-flash-next on Hugging Face. Documented clients are Codex over the Responses API and Claude Code over the Messages API. Sessions can be kept, forked and joined, and there is a classify endpoint. Configuration covers cache modes, a KV pool, disk cache, 1M context and an API key. AGENTS.md and llms.txt exist for agent-driven deployment.

A separate preview image serves RecursiveSI's Flint, a 40B mixture-of-experts model with about 2B active parameters under Apache-2.0. It downloads about 21 GB of int4 weights on first start:

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_SERVE_MODEL=flint \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.18.14-flint-preview

Limits

The engine runs on exactly one chip and one model family, and there is no fallback. The docs recommend giving it a dedicated machine, with recipes for sharing one. In the Flint preview, prefill slows past roughly 80-90k tokens, and speculation is opt-in (HALOGEN_FLINT_SPEC=1) and greedy-only. The engine licence is in LICENSE.md, whose terms the page does not name. Model weights are licensed separately by their authors. "halogen" and "Peonist" are trademarks of Peonist, LLC, though running the engine and publishing numbers need no permission.