drumih/turbo-fieldfare

What it claims

TurboFieldfare is a model-specific Swift + Metal runtime for the pinned instruction-tuned Gemma 4 26B-A4B checkpoint. It does not wrap MLX or llama.cpp. The headline is roughly a 2 GB RAM working budget. The validated target is an 8 GB M2 MacBook Air. The page cites a "measured result" as a reference point but the fetched text carries no tokens/s figure. Throughput numbers live in the linked benchmark and community-results pages.

How it works

Per transformer layer:

flowchart TD
  A[Metal: attention + router<br/>from resident weights] --> B[CPU: top-8 expert IDs<br/>planned against 16-slot LFU]
  B -->|hit| D[Routed experts in cache]
  B -->|miss| C[Parallel pread from SSD<br/>into Metal-visible buffers]
  C --> D
  A --> E[Metal: shared-expert branch<br/>runs during reads]
  D --> F[Combine shared + routed]
  E --> F

• Router and reads. The router picks the top-8 routed experts. The CPU fills cache misses with bounded parallel pread calls, and the shared-expert branch computes while those reads run.
• Prefill. Prefill runs in chunks of up to 128 tokens, so one fetched expert serves several rows. Decode then runs one token at a time.
• Quantization. Embedding, attention, shared-expert and routed-expert weights use 4-bit MLX affine quantization. The router is 8-bit.
• KV cache. The KV cache is FP16. The 25 sliding-window layers use bounded circular storage and the 5 full-attention layers use linear storage. Decode attention uses exact split-K/V.
• Vision. The vision tower releases routed experts while it runs. Image rows attend bidirectionally inside the sliding-window layers.
• Installer. The installer streams byte ranges from the pinned Hugging Face revision and repacks them directly into the .gturbo layout. It never stages a full checkpoint or shard. An install is accepted only after its manifest and file hashes validate.

How to use it

Requirements:

• Apple Silicon (arm64 only)
• macOS 26 with Metal 4
• Xcode 26 and Swift 6.2+
• About 15 GB of download and 14.3 GB installed
• The vision pack adds about 1.1 GB and needs an M2 or newer; M1 is text-only

git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac

Install via CLI. Add --resume to continue an interrupted download, or use --verify-install --input-gturbo ... to check an install:

swift run -c release TurboFieldfareRepack \
  --output scratch/gemma4.gturbo \
  --overwrite

Run from the CLI. Messages go in a JSON array of {"role","content"} objects. --max-new defaults to 1,024. Other flags include --expert-cache-slots, --expert-cache-policy, --prefill-chunk-tokens and --rdadvise. Timing goes to stderr; --quiet suppresses it.

swift run -c release TurboFieldfareCLI \
  --model scratch/gemma4.gturbo \
  --messages-file messages.json

Run the server. It listens on http://127.0.0.1:8080/v1 and supports Chat Completions, streaming, function tools (the client authorizes and executes calls) and single-prefix prompt reuse.

swift build -c release --product TurboFieldfareServer
.build/release/TurboFieldfareServer \
  --model scratch/gemma4.gturbo

Sampling defaults are temperature 0.2, Top-K 64 and Top-P 0.95. Set temperature to 0 for greedy decoding.

Licence. The code is Apache 2.0. The weights are not included and remain under their source terms.

Limits

• One model, one checkpoint. It runs only the pinned Gemma 4 26B-A4B instruct model. Audio and video are out of scope, and older macOS or Metal versions are unsupported.
• One model process at a time. Only one model-owning process (app, CLI, server or tests) should run at once. The author advises checking memory_pressure -Q and closing memory-heavy apps before a run.
• Server security. The server is marked experimental and has no authentication or TLS, so keep it on loopback.
• Tools. The app and CLI do not expose or execute tools.
• Image prompts. Image prompts need at least 16 expert-cache slots because they always prefill chunked. --image works for single-turn only; multi-turn images go in the messages file.
• One active conversation. Only one conversation's KV cache stays resident, so continuing another chat discards it.
• Throughput varies. The page says prompt length, output length, page-cache state and hardware all move throughput. It also notes the model may repeat itself or answer incorrectly.
• Untested hardware and platforms. iPhone/iPad support and benchmarks on more Macs (e.g. the 16 GB M4 Mac mini) are listed as future work.
• Provenance. This is an independent project, not affiliated with Google. The author describes it as possibly "not the most practical."