Skip to content

29 results for “Speculative decoding”

ListGrid
What you seeeverything
RecentOldestAddedA–Z
All types29Papers20Repos5Articles4
“Speculative decoding”
arxiv.orgpaper🛡️ Security

TextSeal2: Distortion-Free LLM Watermarking with Copula Mixing and Memoryless Races

TextSeal2 is a distortion-free LLM text watermark that merges two or more keys into one virtual key and couples it to fresh noise through AutoMark's Gaussian copula. Under speculative decoding it verifies keyed drafts with an extra exponential race, so output matches the target model and acceptance rate is unchanged. On three LLMs at a 10^-3 false-positive rate, it detects 8 to 22 points more 100-token answers than TextSeal.

TextSeal2TextSealAutoMark+2
Gumbel-max schemespeculative decodingGaussian copula+1

AI Security· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper🧠 AI

TextSeal2: Distortion-Free LLM Watermarking with Copula Mixing and Memoryless Races

TextSeal2 is a distortion-free watermark for LLM output that keeps its signal under speculative decoding. It merges two or more keys into one virtual key coupled to fresh noise through AutoMark's Gaussian copula, and verifies drafts with an extra exponential race. On three LLMs it detects 8 to 22 points more 100-token answers than TextSeal at a 10^-3 false-positive rate.

LLMspeculative decodingGumbel-max watermarking+1
TextSeal2TextSealAutoMark+2

research, inference· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper🧠 AI

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

This paper measures the energy cost of speculative decoding in SGLang with CUDA graphs on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. Savings depend on drafted-token acceptance, which is lower on conversation than code and lower on Romansh than English. Savings shrink as batch size grows, and block drafters can push verification into the compute-bound regime, where speculation raises energy per token.

SGLangCUDALLM+4
NVIDIAarXivRomansh

inference, research· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper🧠 AI

Multi-Token Prediction Done Right

Soft Multi-Token Prediction (Soft-MTP) is a one-line change to multi-token prediction (MTP) pre-training. The authors show the MTP cross-entropy gradient splits exactly into a KL self-distillation term and a noisy residual, and a straight-through "KL gradient-recovery trick" keeps only the KL term. From 167M to 1.2B parameters it gives lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy than standard MTP, with no added parameters or overhead.

LLMTransformer
DeepSeekMeta

research, training· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper🧠 AI

Evaluating the Applicability of LLM Inference Optimizations on Apple Silicon

This October 2026 arXiv paper evaluates four LLM inference optimization families on Apple Silicon: KV-cache pruning, speculative decoding, ported GPU kernels and post-training quantization. Tests run on four Apple machines with a single-request decode workload, plus matched RTX 4090 runs. KV-cache pruning reaches 1.75x throughput on Apple, all tested speculative decoding methods reduce throughput, and ported kernels give up to 1.20x.

MetalCUDALLM
AppleNVIDIAarXiv

inference, research, evals· Added 24 Oct 2026, published 22 Oct 2026

arxiv.orgpaper🧠 AI

Large Flow Language Models

Lafa (Large flow language model) is a blockwise continuous flow language model scaled to 8B parameters, with causal attention on the prefix and bidirectional, asynchronous flow within each block. It is adapted from pretrained autoregressive models and trained at 0.5B, 7B and 8B with under 30B tokens. A self-speculative decoding scheme verifies flow drafts autoregressively in one model, giving 2.54x the throughput of an AR model at comparable quality. Flow maps allow one- and few-step generation.

flow matchingdiffusion modelTransformer+1
Lafa

research, models· Added 22 Oct 2026, published 20 Oct 2026

arxiv.orgpaper🧠 AI

Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding

This paper shows that training-free relaxed (lossy) verification in speculative decoding causes length inflation: completions get longer, mostly through extended reasoning traces, which offsets gains in accepted tokens per round. Across twelve target-drafter pairs, relaxed rules accept 1.34x as many tokens per round but produce 1.54x longer completions. Rules that cut verifier rounds usually lose substantial accuracy, so the authors argue completion length and verifier rounds must be reported.

LLM
arXiv

research, inference, evals· Added 20 Oct 2026, published 16 Oct 2026

arxiv.orgpaper🧠 AI

DLite: History-free Speculative Decoding for Agentic AI

DLite is a history-free speculative decoding framework for agentic LLM workloads. It reuses the target model's contextual hidden states and predicts locally without keeping the draft model's historical KV cache, which avoids the up to 80% acceptance-length drop seen with long, multi-turn contexts. A lightweight sequential head restores intra-block causal dependencies, and a two-stage training strategy is used. It reaches 2-6 acceptance length and beats DSpark by up to 70%.

LLM
DLiteDSparkarXiv

inference, research· Added 20 Oct 2026, published 16 Oct 2026

huggingface.coarticle🧠 AI

Welcome Beam: a 501B open model you can own, customize, and scale

Announces Reflection's Beam-1.0-501B-A23B, an Apache 2.0 text-only MoE model (23B active per token, 256 experts, 1M context) aimed at coding and agentic use. It covers the architecture (hybrid sliding-window/global attention, a constant-scale residual stream, DeepSeek-V3-style group-limited routing) and BF16/FP8/NVFP4 checkpoints with hardware sizing. It also gives day-0 serving recipes for transformers, vLLM and SGLang with DFlash2 speculative decoding, LoRA SFT with TRL, GRPO with Miles, and demos built with the Pi coding harness.

TransformersvLLMSGLang+12
BeamReflectionDeepSeek+15

models, inference, training· Published and added 16 Oct 2026

arxiv.orgpaper🧠 AI

SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

SP$^2$ec (Single-Peak SPeculative decoding) is a training-free bandit algorithm that adapts the draft configuration of self-speculative decoding per prompt for vision-language models. It relies on the single-peak throughput pattern seen in SWIFT, KnapSpec, SpecVLM and the authors' cosSim method. It minimizes a new wall-time regret measure and has theoretical guarantees of near-optimal configuration with negligible exploration cost. On Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B it improves over the original self-speculative methods by up to 11% on visual and textual tasks.

vision-language modelQwenMistral
SP$^2$ecSWIFTKnapSpec+5

research, inference, multimodal· Added 14 Oct 2026, published 13 Oct 2026

arxiv.orgpaper🧠 AI

ReplaySSM: Cache SSM Inputs, Not State

ReplaySSM represents state-space model state as a periodically updated checkpoint plus a buffer of recent SSM inputs, separating output computation from checkpoint updates. This avoids state writes on most decoding steps (roughly halving dominant state memory traffic), makes speculative-decoding rollback a buffer-pointer operation, and lets a low-precision checkpoint serve outputs while a higher-precision one drives recurrent updates. Implemented in vLLM for Mamba-2 and Gated DeltaNet, it gives 1.20–1.47x decoding speedup at batch size 256 across four hybrid models and about 1.8x the throughput of vLLM speculative decoding on two 120B-class models.

vLLMMamba-2Gated DeltaNet+1
ReplaySSM

inference, research· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

Adaptive DHiSparse extends HiSparse (host-memory KV storage that fetches only selected entries) to draft-model speculative decoding methods DSpark and DFlash2. It protects the required working set, keeps extra history while GPU memory is spare, and recovers the rest by exact fused retrieval. Queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 reaches 793 output tokens/s per decode GPU (1.45x HiSparse). With the same latency bounds it sustains up to 4x the concurrency. On DeepSeek-V4.1-Flash at 500K context, it delivers 1.85x the output of resident serving with DSpark.

Adaptive DHiSparseHiSparseDSpark+4

inference, research· Added 12 Oct 2026, published 9 Oct 2026

arxiv.orgpaper🧠 AI

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

The paper formalizes speculative decoding as a Markov reward process and derives the Expected Decoding Rounds (EDR) objective, a hyperparameter-free training loss equal to the expected number of decoding rounds, addressing cross-round coupling that prior block-local surrogate objectives ignore. It derives an exact temporal-difference gradient for unbiased stochastic optimization and an offline evaluator for comparing drafters on shared rollouts without running speculative decoding. Finetuning the DSpark and DFly drafters with EDR improves mean accepted length and beats prior training objectives across nine benchmarks in math, code, and chat.

DSparkDFly

research, training· Added 8 Oct 2026, published 7 Oct 2026

arxiv.orgpaper🛡️ Security

Secure Speculative Decoding for Large Language Models

This arXiv paper presents the first systematic study of security implications of speculative decoding, a technique that speeds up LLM inference using a smaller draft model to propose tokens verified by the target model. The authors find a security-utility asymmetry: lossy speculative decoding methods raise jailbreak and prompt-injection attack success rates much faster than they degrade utility, with security degradation traced to early tokens generated by the draft model. They propose SecureSD, which applies stricter verification to early-position draft tokens, improving security while preserving inference efficiency and utility on security and utility benchmarks.

LLM
SecureSD

AI Security· Added 7 Oct 2026, published 6 Oct 2026

arxiv.orgpaper🧠 AI

Learning What to Imitate: Entropy-Aware Distribution Mixing

The paper proposes Entropy-Aware Mixing, a knowledge-distillation method where a small student model's per-token distribution is dynamically interpolated with a stronger teacher model's distribution, gated by the student's own predictive entropy, to avoid "confident conflicts" where the student must imitate low-probability continuations despite being confident in a different one. The authors implement both convex and geometric interpolations under two training regimes — offline trace generation via speculative decoding followed by SFT, and on-policy forward-KL distillation — and show it stabilises distillation, improving in- and out-of-distribution math reasoning while better preserving general capabilities than fixed-teacher supervision, with offline traces favouring concave entropy schedules and on-policy training favouring linear/convex ones.

training· Added 6 Oct 2026, published 5 Oct 2026

latent.spacearticle🧠 AI

[AINews] not much happened today

Daily AINews digest (Latent Space) covering an October 2026 news cycle: OpenAI's cheaper/faster GPT-6.1 Sol and Anthropic's Claude Sonnet 5.5 reshaping Agent Arena's cost-performance frontier, a wave of decision-model releases (Cloudflare Clef, llama.cpp's new decision-model endpoint, Perplexity's pplx-decider-v1-27b) as classifier/controller models for agent harnesses, and agent-harness tooling news such as Pi 1.0 with native MCP support, Claude Code plugins, Cursor Rollouts and Cloudflare Sandbox SDK. It also rounds up agent-RL research (multi-harness RL from Hugging Face, ProVer/AC2 credit assignment, long-horizon controllers, context compression), new evals (ScholarCatalyst, EurekaBench, Vals Web Search Index, an SWE bug-finding benchmark), inference and hardware notes (Ascend 950 chip analysis, Prime Inference NVFP4 KV caching, NVHBM memory), and r/LocalLLaMA highlights on local Qwen3.8-27B inference and MTP speculative decoding in llama.cpp.

Model Context ProtocolClaude Codellama.cpp+9
GPT-6.1 SolGPT-6 SolGPT-6 Astra+41

agents, evals, models, inference· Published and added 3 Oct 2026

huggingface.corepo🧠 AI

Tencent Hy4 preview (open-weight MoE + Sherry quantization)

Hy4 preview is Tencent's new flagship Mixture-of-Experts language model: 770B total parameters with 49B activated per token across 78 layers (256 routed experts + 1 shared expert, top-8 routing), plus a native 10B multi-token-prediction layer for speculative decoding. It introduces Gated DeepSeek Sparse Attention with IndexCache for cross-layer index reuse and identity Hyper-Connections (iHC) on the residual path, and in blind internal evaluations on engineering tasks it edges out GLM-5.3 and Kimi K3. Weights (including an FP8 variant) are released under Apache 2.0 on Hugging Face with vLLM/SGLang deployment recipes and AngelSlim-based quantization/compression tooling.

Hugging FacevLLMSGLang+4
TencentTencent Hy4 previewHy3 preview+17

models· Added 2 Oct 2026

arxiv.orgpaper🧠 AI

EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures

EdgeAgent is a cross-layer LLM inference system for running multi-agent workloads on edge devices with CPU-GPU unified memory architectures such as Apple M-series SoCs. It introduces zero-copy UMA-aware tensor parallelism to saturate both CPU and GPU, a dynamic speculative-decoding draft-budget scheduler adapting to sequence predictability, and an asynchronous suspend-and-yield mechanism that evicts stalled agents during tool-call latencies. On an Apple M4 SoC it reports 1.29x speedup from UMA-aware execution alone and 1.77x with full agent-aware scheduling, versus batched speculative decoding.

speculative decodingtensor parallelism
Apple M4

inference, agents· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper🧠 AI

Execution-Path Qualification and Realized Costs in Speculative Decoding on Consumer Systems

An arXiv preprint that empirically tests speculative decoding on consumer hardware (Intel Arc B570, NVIDIA RTX, Apple M4-class systems), distinguishing between 'execution-path qualification' (whether a speculative predictor's output matches the target model's exact generation trajectory) and the realized latency/cost of running speculation. The authors find that a target-only, draft-free predictor matches greedy output on most repeat-stable prompts but does not guarantee qualification or identify a unique cause of divergence, and that newly qualified RTX serialization and other configurations (358H, M4 Guard) are often slower than target-only decoding once real costs are counted. The paper's conclusion is that these results are diagnostic engineering findings specific to the tested hardware/software paths, not evidence of a portable decoding speedup.

CUDA
B570RTXApple M4

inference· Added 5 Oct 2026, published 2 Oct 2026

arxiv.orgpaper🧠 AI

DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair

DRelay is a speculative decoding technique for LLM inference that repairs parallel draft-token selections before target-model verification, using a global reader over the entire draft block plus a causal selector to decide whether to keep or replace each candidate token, extending the accepted prefix length. Trained jointly with the draft backbone via a combined candidate-support and position-weighted repair loss, it is evaluated on eight benchmarks on an H800 GPU and, under SGLang serving, improves end-to-end decoding speedup substantially over DFlash, Domino, and DSpark baselines.

SGLang
DRelayDFlashDomino+1

inference· Added 2 Oct 2026, published 1 Oct 2026

arxiv.orgpaper🧠 AI

AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines

AgSpec is a retrieval-based speculative decoding framework built for coding-agent pipelines. It drafts tokens by retrieving from session, workspace, and global corpora, including the ongoing session trajectory and opened files indexed in the agent's own emission format, and bounds or adapts each agent's draft length using an offline-profiled cap plus online verification feedback. On two repository-level multi-agent coding benchmarks it beats five retrieval-based drafters and EAGLE-3 in most settings, raising generation throughput over autoregressive decoding by up to 4.37x at batch size 1 and 4.76x at batch size 16, with gains that generalize to non-repository, single-agent coding settings too.

EAGLE-3

inference, coding-agents· Added 2 Oct 2026, published 1 Oct 2026

huggingface.corepo🧠 AI

DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters, 1M-token context, and a novel Causal Encoder-Decoder (CED) architecture where the decoder's global KV cache is projected from encoder hidden states rather than recomputed per layer, activating only 8B/16B parameters per token during prefill/decode. It introduces Compressed Sparse Attention 2 (CSA2) with FP4 KV caching to cut the global KV cache footprint to 890 bytes/token (~1/4 of DeepSeek-V4-Flash, ~1/437 of DeepSeek-V1), alongside Engram conditional memory, DSpark speculative decoding, and a continuously controllable reasoning-effort setting (1–100). The card documents pre/post-training recipes (45T tokens, SFT→RL→on-policy distillation with synthetic agent environments) and reports results across agentic and code benchmarks (Terminal-Bench 2.1/3.0/4.0, DeepSWE v1.1, GPQA Diamond, Chartography, ZeroBench, AutomationBench), plus deployment instructions for Transformers, vLLM, SGLang, and Docker.

TransformersvLLMSGLang+5
DeepSeek-V4.1-FlashDeepSeek-V4-FlashDeepSeek+16

models, multimodal, inference· Added 30 Sep 2026

github.comrepo🧠 AI

vLLM v0.30.0

Release notes for vLLM v0.30.0, a major update to the open-source LLM inference/serving engine spanning 762 commits. Highlights include new model support (DeepSeek-V4.1-Flash, GLM-5.3-Flash, Kimi K3, Qwen3.8-Flash-Next, K2-Horizon, Cohere Compass and others), a "Fast Start" GPU weight-cache daemon for near-instant engine restarts via CUDA IPC, Gumbel-max watermarking for generated text, a HiSparse host-memory tier for sparse-MLA KV offload, Model Runner V2 improvements (dual-batch overlap, speculative decoding under pipeline parallelism), expanded quantization support (NVFP4, W4A16 DSA, AutoRound), and hardware-specific tuning across NVIDIA, AMD ROCm, Intel XPU and CPU backends.

vLLMCUDAFlashInfer+9
DeepSeek-V4.1-FlashDeepSeek-V4-Flash-Vision-ExpGLM-5.3-Flash+16

inference, models· Added 28 Sep 2026

arxiv.orgpaper🧠 AI

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

ASPIRE is a non-synchronized batched self-speculative decoding framework for long-context LLM inference, where requests independently choose when to verify instead of sharing a global draft-verify schedule. It combines a unified mixed forward pass, a per-request online speculation scheduler using acceptance-rate estimates and a batch-aware cost model, and an intra-draft refresh layer that periodically applies full attention to reduce staleness. Across three models and five reasoning/long-context benchmarks it reports 1.70-4.58x decoding throughput speedup over autoregressive baselines, ~27% better than prior self-speculative baselines.

ASPIRE

inference, research· Added 26 Sep 2026

arxiv.orgpaper🧠 AI

How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method

Speculative Uncertainty (SU) repurposes speculative decoding in reverse: a small open-weight draft model scores an agent's already-generated trajectory in a single forward pass, producing a failure-likelihood signal from output tokens alone without needing logits, weights, or repeated sampling. Phase-aware features separate reasoning and action spans and are calibrated against a verifiable objective to drive a pre-execution veto gate. On software engineering agents built on Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, the gate cuts execution error rate by 6-8 percentage points and token cost by 14-19%, transferring to out-of-distribution benchmarks without retraining.

speculative decoding
Qwen3-Coder-480BClaude 3.5 Sonnet

coding-agents, research· Added 26 Sep 2026

huggingface.coarticle🧠 AI

LFM2.5-VL-3B-DSpark (Liquid AI)

Liquid AI releases LFM2.5-VL-3B-DSpark, a 280M-parameter attention-only draft model (4 layers, block size 9, +8.9% parameters) for speculative decoding of the LFM2.5-VL-3B vision-language model. It reports decode speedups of 2.30–3.13x on MLX (M5 Max), 1.57–2.14x on llama.cpp (M3 Ultra) and up to 2.66x on H100, with end-to-end gains of 1.30–2.62x, evaluated on six MMSpec vision tasks. It explains why prefill and vision encoding limit end-to-end gains (Amdahl's law) and gives launch commands for SGLang, llama.cpp and MLX-VLM.

SGLangllama.cppMLX+7
Liquid AILFM2.5-VL-3BLFM2.5-VL-3B-DSpark+5

inference, models, multimodal· Added 26 Sep 2026, published 24 Sep 2026

machinelearningmastery.comarticle🧠 AI

The Roadmap to Mastering LLM Inference Optimization

A structured guide walking through the layered techniques used to make LLM inference faster and cheaper: understanding the compute-bound prefill vs. memory-bandwidth-bound decode phases, KV caching with PagedAttention (as implemented in vLLM) and prefix caching, static/dynamic/continuous batching, attention optimizations (MQA, GQA, FlashAttention), model compression (GPTQ/AWQ quantization, 2:4 structured sparsity, knowledge distillation), speculative decoding for latency-critical single-request scenarios, and scaling via tensor/pipeline parallelism and prefill-decode disaggregation. It ties each technique to the specific bottleneck it addresses, making it a practical map for choosing optimizations based on latency vs. throughput requirements.

vLLMTensorRT-LLM
PagedAttentionFlashAttentionGPTQ+2

inference· Added 28 Sep 2026, published 21 Sep 2026

github.comrepo🧠 AI

Nehanth/pooled

Pooled runs one open model (Qwen 3.6 35B MoE, a 27B dense hybrid, smaller Qwen3/SmolLM) across devices in a browser "room". Each tab holds a slice of layers and runs them on its own GPU with custom WGSL kernels. Hidden states pass over WebRTC, and speculative decoding is checked against plain decoding by golden tests. It also ships a Code-mode agent loop with a sandboxed preview, and an npx CLI bridge (@pooled/cli) that exposes the room as an OpenAI Chat Completions/Responses and Anthropic Messages endpoint with grammar-constrained tool calling, so Codex CLI, Claude Code and opencode can use it. The README includes latency-injected benchmarks, a comparison with exo, Petals, llama.cpp RPC and WebLLM, and a threat model.

WebGPUWGSLWebRTC+17
PooledNehanth NarendrulaQwen+14

inference, coding-agents· Added 16 Oct 2026, published 1 Sep 2026

github.comrepo🧠 AI

patchy631/time-to-first-token

A 10-week, 30-minutes-a-day roadmap (GitHub repo "time-to-first-token") that walks engineers through building one continuously-evolving OpenAI-compatible inference service rather than disconnected tutorials: standing up vLLM and SGLang, reading the PagedAttention/RadixAttention source, instrumenting Grafana dashboards for TTFT/inter-token latency/throughput/queue depth, load-testing past 1000 concurrent requests with GuideLLM, benchmarking FP8/INT4/AWQ quantization and speculative decoding, configuring KV cache eviction, deploying prefill/decode disaggregation on Kubernetes, and building a cost/latency/quality router. It closes with a checklist of common public-benchmark mistakes (sweep request rate, report p50/p95/p99, separate TTFT from inter-token latency, pin versions) and pointers to Stanford CS336, MIT EfficientML.ai, vLLM Office Hours, and GPU MODE for further study.

vLLMSGLangDocker+9
RadixAttentionPrometheusStanford CS336+17

inference, evals· Added 2 Oct 2026, published 2 Aug 2026