arxiv.orgpaperFlash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
Flash-dLLM is a training-free inference acceleration framework for diffusion large language models (dLLMs) that tackles GPU memory I/O bottlenecks in KV-cache-enabled dLLM inference via an I/O-aware fused KV-cache kernel, combined with a draft-and-verify decoding strategy where the same dLLM acts as both drafter and verifier. It reports 5.1x and 11.0x speedups over the prior strongest baseline Elastic-Cache on GSM8K and HumanEval respectively, while preserving generation quality and scaling to longer sequences and larger batches.
TL;DR A training-free KV-caching and parallel-decoding framework that gives diffusion LLMs up to 11x faster inference without an auxiliary drafter model.
Organisations named here
- GSM8K
- HumanEval
Technologies named here
- Flash-dLLM
- Elastic-Cache
- diffusion LLM
Techniques named here
- KV caching for diffusion LLMs
- I/O-aware fused KV-cache kernel
- KV-cache-driven draft-and-verify decoding
- Parallel decoding
Related in inference
arxiv.orgpaperMinimally Invasive Steering of Language Models
x.comthreadSamuel Zeng (@SamuelZengML) on X
huggingface.coarticle