Transformers now runs llama.cpp quants

What it claims

Transformers can load GGUF checkpoints from the Hub with from_pretrained(..., gguf_file=...) and keep the weights packed on Apple Silicon, running them through ggml's Metal kernels distributed via the kernels library. The post says decode speed on an M2 Max is "close to llama.cpp" for three Qwen3.5 checkpoints: a small dense model, a larger dense model and an MoE. The authors still recommend llama.cpp when efficient local inference is the priority. They pitch the integration for other work done in PyTorch: hooks and forward-pass edits, running existing evaluation workflows on quantized checkpoints, checking a GGUF conversion against the original weights, custom logits processors and stopping criteria, and fine-tuning after dequantizing.

How it works

Loading picks a path automatically, and each fast path has a silent fallback:

flowchart TD
    A[from_pretrained gguf_file] --> B{Compatible quant kernel?}
    B -- no --> C[Dequantize: more memory]
    B -- yes, MPS --> D[Packed weights + ggml Metal layer kernels]
    D --> E{ggml-org/ggml-attn fetchable?}
    E -- yes --> F[ggml-attn attention]
    E -- no --> G[sdpa, with warning]

• Kernels. Four kernel packages come from ggml. A fifth, a top-k kernel, targets MoE routing. Everything runs in eager mode, with no torch.compile, so there are no compile pauses and no recompilation when input shapes change.
• Changes to generate. These apply to all models, not only GGUF:
• #48814 drops the all-ones attention mask at the start of generation when a decoder-only input has no padding.
• #47975 copies the stopping decision asynchronously and reads it on the next step, so the CPU keeps scheduling work. Any extra token generated past the stop condition is trimmed. Streaming uses the same mechanism.

Evidence

Setup

• Hardware and software: MacBook Pro M2 Max, 32 GB, macOS 26.6, PyTorch 2.12.1, kernels 0.17.0, plugged in.
• llama.cpp baseline: llama-bench build 5f55650a7 (b10200, ggml 0.18.0 Metal) with -p 0 -n 128 -r 3, reporting tg128 decode-only throughput.
• Transformers measurement: 128 tokens from a 12-token prompt, best of three warmed runs, with prefill included.

Caveats on comparability

• The two columns do not measure the same thing: one excludes prefill, the other includes it.
• The script sleeps 90 s between runs because back-to-back runs "decay by 10% or more".

Ablations The post also has two ablations:

• Layer kernels on vs. off, with the quantization kernel enabled in both.
• Generation-loop changes isolated, with all kernels enabled.

How to use it

Requirements:

• An Apple Silicon Mac.
• A PyTorch version covered by the published kernel builds, usually the two latest releases.
• Transformers from main.

pip install -U "git+https://github.com/huggingface/transformers.git" kernels

To force standard attention, pass attn_implementation="sdpa".

Serving uses <repo>:<file>.gguf syntax and exposes an OpenAI-compatible API:

pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels
transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"

Reasoning is controlled with --reasoning on|off|auto. The default, auto, follows the chat template.

For fine-tuning, pass quantization_config=GgufConfig(dequantize=True) with dtype=torch.bfloat16.

Quantization advice from the post: start with Q4_K_M, and move to Q5_K_M or Q6_K if you have spare memory.

Limits

• No figures in the text. Speed results appear only in charts, so you need the page itself for the numbers.
• Silent memory cost. Without a compatible kernel, the loader dequantizes the model, which uses more memory and does not raise an error. Check memory use before trusting the result.
• Padding and batching. Padded batches miss the mask shortcut and can be slower. generate_batch on MPS is future work.
• Text only. Other modalities and non-GGUF models reusing these kernels are described as a direction, not a shipped feature.
• Requests. Users who want other architectures supported are asked to open an issue.