patchy631/time-to-first-token

What it claims

The roadmap is 50 sessions of 30 minutes each, spread over 10 weeks with buffer days. Every session feeds one service, on the argument that "seventeen disconnected experiments" spend most of their time on setup. It assumes you know Python, transformer architecture and the command line. It does not assume prior serving, Kubernetes or CUDA experience.

How it works

Five ordering decisions drive the schedule, and they are the main reason to read it over a generic link list:

• Roofline first (week 1). Each later optimization is framed as a move on that one plot:
• Quantization attacks memory-bandwidth-bound decode.
• Continuous batching raises arithmetic intensity.
• Speculative decoding spends cheap FLOPs to cut sequential memory loads.
• Disaggregation exists because compute-bound prefill and bandwidth-bound decode compete for one GPU.
• The week 1 deliverable is a hand-derived arithmetic-intensity figure for your own model.
• Measurement before tuning. Grafana dashboards come in week 3 and load testing in week 5, both before any optimization knob.
• PagedAttention and continuous batching are read, not reimplemented. You read vLLM's block manager and V1 scheduler until you can explain from the code why memory waste falls under 4%.
• The router is paired with unit economics. Building it forces a dollar figure and a latency budget into code.
• Edge runtimes are optional and last. ONNX Runtime, TensorRT-LLM and WebLLM share almost no operational surface with the datacenter stack.

Some weekly deliverables are more specific than the summary suggests:

• The week 5 sweep must report p50, p95 and p99, and be cross-checked by two independent tools.
• Quantized variants are judged on throughput and a quality proxy against the FP16 baseline.
• Week 7 explicitly includes negative results.
• Kubernetes autoscaling in week 8 is driven by queue depth, not CPU.
• The router includes per-request token budgeting, with cost shown on the dashboard.

How to use it

Hardware and cost:

• A 7-8B model needs about one 24 GB GPU. Rent it by the half hour and shut it down between sessions.
• No GPU is needed for all of week 1, most of week 3, the week 6 lecture days, or the week 9 reading.
• An H100 is needed for only two sessions: the 1000-concurrent test in week 5, and week 8 disaggregation if you run it for real.

Install:

pip install vllm
pip install guidellm
pip install sglang

You also need Docker for Prometheus and Grafana, and kind or a small managed cluster for week 8.

Progress tracking has three options:

1. index.html or GitHub Pages. Progress lives in browser localStorage. A 12-character sync link or a JSON export moves it between devices.
2. A GitHub issue from the progress template. This syncs across devices and keeps a dated history.
3. Ticking the README in a fork.

The benchmark checklist adds rules beyond those in the summary:

• Report exact input and output token lengths.
• Report the knee, not the preemption/saturation collapse.
• Avoid uniform-length synthetic traffic.
• Account for tokenizer differences before comparing tokens per second.

Licence: CC BY 4.0.

Limits

• All performance figures quoted in the roadmap are the sources' own claims and were not independently reproduced. Re-derive GPU rates and per-token costs against your own hardware.
• Links may rot:
• vLLM docs change between versions, and its blog uses two URL shapes. Search the exact title if a link 404s.
• Video timestamps are approximate.
• MIT 6.5940 was not offered in Fall 2025. Use the 2023 or 2024 recordings.
• For GPTQ, AWQ, SmoothQuant, Medusa, H2O, SnapKV and FlashAttention, the roadmap points to surveys rather than primary links. It warns against guessing arXiv IDs.
• localStorage may be blocked in private or sandboxed browsing. The page warns you, and ticks then last only for that visit.