The Roadmap to Mastering LLM Inference Optimization

What it claims

This is an introductory survey, not a benchmark study. Its organizing claim is that decode, not prefill, is the binding constraint in generation. Decode is limited by memory bandwidth, so the most useful optimizations reduce bytes moved per token. The two phases map onto separate metrics:

• Time-to-first-token (TTFT) reflects prefill.
• Tokens per second (TPS) after the first token reflects decode.

Improving one does not necessarily improve the other. The article says to pick techniques by whichever metric your use case depends on, then add them in order of impact after profiling your actual workload.

How to use it

These are the decision-relevant specifics the page gives beyond naming each technique.

Changes you can make without retraining

• FlashAttention is described as a drop-in replacement with identical mathematical output. The page calls it one of the easiest wins.
• PagedAttention and continuous batching are already the defaults in vLLM. Continuous batching is also the default in TensorRT-LLM. If you use either runtime, you likely already have these.
• Prefix caching pays off when requests share a prefix, such as long system prompts, RAG documents or few-shot blocks.

Changes that require training or a different model

• MQA and GQA only help if the model was trained or fine-tuned with them. You cannot switch them on at serving time. The page says MQA costs a small amount of accuracy, and GQA keeps more of the original multi-head attention's capacity.
• Distillation produces a new, smaller student model. The page says that when latency is the main constraint, distillation often beats compressing the original model.

Memory sizing figures the page gives

• A 7B model needs about 14 GB in FP16 and about 3.5 GB in INT4. That gap can change which hardware tier can serve the model at all.
• In 16-bit, the KV cache for a 7B model can take "several gigabytes per request" at moderate sequence lengths. It grows with both batch size and sequence length.
• GPTQ and AWQ are named as the methods that make 4-bit quantization practical.

Sparsity

• 2:4 structured sparsity (two zeros in every four values) has hardware support on NVIDIA Ampere and later GPUs. The page cites "up to 2x" speedup on compatible operations.

Batching

• Dynamic batching is tuned with a maximum batch size and a timeout. A shorter timeout lowers latency but produces smaller batches. A longer timeout raises throughput but adds queuing delay.
• Even with dynamic batching, short requests stay blocked behind long ones once a batch starts. Continuous batching removes this.

Speculative decoding

• Accepted tokens are guaranteed to match what the full model would have produced, so output quality is unchanged.
• The speedup depends on how often the draft model agrees with the full model.
• It helps most for single-request, interactive work such as chat. It helps little for high-throughput batch workloads, where batching already keeps the GPU busy.

Parallelism

• Tensor parallelism splits each layer across GPUs, which cuts per-GPU weight memory and lowers latency.
• Pipeline parallelism assigns consecutive layers to different GPUs. It suits very large models and high-throughput workloads. It suffers from pipeline bubbles (idle GPUs), which microbatching reduces.
• Prefill-decode disaggregation runs the two phases on separate hardware pools. It stops long prefills from blocking decode, which matters more as context lengths grow.

Limits

• The page contains no benchmarks, measured baselines or hardware settings. The "up to 2x" sparsity figure and the 7B memory figures are general claims, not experiments.
• There are no commands, configuration flags or code. Each step links to a separate article for depth.
• The accuracy cost of quantization (called "minimal") and of MQA (called "small") is never quantified.
• The summary table referenced at the end is not present in the fetched text.