Evaluating the Applicability of LLM Inference Optimizations on Apple Silicon

What it claims

MacPaw AI&Research (October 2026) tests whether four NVIDIA-validated inference optimization families still help batch-one local decoding on Apple Silicon. Their conclusion is that a speedup belongs to the combination of technique, runtime, hardware and workload, not to the technique alone. Beyond the summary's headline figures, the overview table gives these per-family results:

• Prefix reuse: persistent prefix reuse cuts wall time by 57% at turn 9.
• MoE Gather: the ported kernel gives +3 to 12% on every device.
• FlashInfer split-KV attention: +20% on M4 Pro but −3% on M2.
• Marlin W4A16 GEMM: helps only at batch size above 1 and is negligible at M=1.
• Quantization: MLX native affine and DWQ quantization give the best combined trade-off. QuaRot does not pay off without low-precision activation kernels.
• Speculative decoding: none of Classic, N-gram, Cascade-n-gram, EAGLE-3, DFlash or DDTree achieves a net speedup on M2 Pro or M3 Ultra.

How it works

The KV-cache experiments run Qwen3-4B-Instruct-2507 and LFM2.5-2.6B in bfloat16 through Transformers and KVPress. Contexts are 512 and 4,096 tokens, outputs are 1, 16, 128 and 256 tokens, and batch size is one. Qwen runs on M2 Pro, M3 Ultra, M4 Pro and an RTX 4090; LFM runs on the three Apple machines only.

Six pruning methods are tested:

• SnapKV, KNorm, StreamingLLM and TOVA at a 0.5 compression ratio.
• ExpectedAttention and FastKVzip at 0.75.

Each speedup is the unpruned time divided by the pruned time on the same device. Timings are medians of three runs after one warm-up. Kernel work ports CUDA mechanisms from Marlin, FlashInfer and MegaBlocks to Metal/MLX and measures them at both kernel and end-to-end level.

The paper also gives a closed-form cache size model, built from attention layers, KV heads, head dimension and sliding-window length. At 32,768 tokens it gives:

• Qwen3-4B: 4,608 MiB (36 global-attention blocks).
• Gemma-3-4B: 756 MiB (29 of its 34 attention blocks are 1,024-token local).
• LFM2.5-2.6B: 512.2 MiB (hybrid).
• Granite-4.0-H-Micro: 292.9 MiB (hybrid).

Evidence

Qwen at 4,096 context and 256-token output. ExpectedAttention reaches 1.75x on M2 Pro, 1.61x on M3 Ultra and 1.72x on M4 Pro, against 0.98x on the RTX 4090. SnapKV, KNorm, StreamingLLM and TOVA give 1.36 to 1.50x on Apple and 0.99 to 1.02x on the 4090. At 512 context, the best Apple result is 1.24x. FastKVzip is below baseline across the whole grid, at 0.28 to 0.96x.

Hybrid LFM2.5-2.6B. Gains are smaller because the 22 short-convolution blocks are not pruned. At 512 context, results range from 0.92 to 1.03x. At 4,096 context and 256-token output they reach at most 1.25x.

Peak memory on M4 Pro. At 4,096 context, Qwen saves 20.4 to 25.6%. At 512 context, pruning raises peak memory by up to 155.5% for Qwen and 48.5% for LFM. The authors attribute this to transient scoring and compaction allocations.

Prefix reuse. In an MLX-LM conversation with Qwen3-4B, wall time falls 33%, 42% and 57% at turns 3, 5 and 9. The saving comes from skipped prefill, not from faster steady-state decoding.

How to use it

The page gives no install commands and no code release in the fetched text. These conclusions follow from the stated results:

• On Apple hardware, pruning pays off with long contexts and long outputs on attention-only models.
• With short contexts, pruning can slow decoding and raise peak memory.
• Persistent prefix caching in MLX-LM is the consistent win for multi-turn chat.

Cache footprints come from these checkpoints: mlx-community/Qwen3-4B-4bit, mlx-community/gemma-3-4b-it-4bit, mlx-community/granite-4.0-h-micro-4bit and LiquidAI/LFM2.5-2.6B.

Limits

• Output quality: the authors do not measure how pruning changes output quality.
• FastKVzip: it needs model-specific pretrained gates, so it is not tested on LFM.
• LFM2.5-8B-A1B: it is excluded because KVPress does not support its architecture and its weights exceed M2 Pro memory.
• Peak memory: it is reported only on M4 Pro.
• Hardware contract: public Metal lacks asynchronous copy, which limits how much of the CUDA kernel designs transfers.
• Truncated text: the fetched page stops early in the speculative decoding section. The speculative decoding, kernel and quantization results above come only from the abstract and overview table, without their per-device numbers or setups.