29 results for “Speculative decoding”

TextSeal2 is a distortion-free LLM text watermark that merges two or more keys into one virtual key and couples it to fresh noise through AutoMark's Gaussian copula. Under speculative decoding it verifies keyed drafts with an extra exponential race, so output matches the target model and acceptance rate is unchanged. On three LLMs at a 10^-3 false-positive rate, it detects 8 to 22 points more 100-token answers than TextSeal.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026🛡️ Security

TextSeal2 is a distortion-free watermark for LLM output that keeps its signal under speculative decoding. It merges two or more keys into one virtual key coupled to fresh noise through AutoMark's Gaussian copula, and verifies drafts with an extra exponential race. On three LLMs it detects 8 to 22 points more 100-token answers than TextSeal at a 10^-3 false-positive rate.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026🧠 AI

This paper measures the energy cost of speculative decoding in SGLang with CUDA graphs on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. Savings depend on drafted-token acceptance, which is lower on conversation than code and lower on Romansh than English. Savings shrink as batch size grows, and block drafters can push verification into the compute-bound regime, where speculation raises energy per token.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026🧠 AI

Soft Multi-Token Prediction (Soft-MTP) is a one-line change to multi-token prediction (MTP) pre-training. The authors show the MTP cross-entropy gradient splits exactly into a KL self-distillation term and a noisy residual, and a straight-through "KL gradient-recovery trick" keeps only the KL term. From 167M to 1.2B parameters it gives lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy than standard MTP, with no added parameters or overhead.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026🧠 AI

This October 2026 arXiv paper evaluates four LLM inference optimization families on Apple Silicon: KV-cache pruning, speculative decoding, ported GPU kernels and post-training quantization. Tests run on four Apple machines with a single-request decode workload, plus matched RTX 4090 runs. KV-cache pruning reaches 1.75x throughput on Apple, all tested speculative decoding methods reduce throughput, and ported kernels give up to 1.20x.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026🧠 AI

Lafa (Large flow language model) is a blockwise continuous flow language model scaled to 8B parameters, with causal attention on the prefix and bidirectional, asynchronous flow within each block. It is adapted from pretrained autoregressive models and trained at 0.5B, 7B and 8B with under 30B tokens. A self-speculative decoding scheme verifies flow drafts autoregressively in one model, giving 2.54x the throughput of an AR model at comparable quality. Flow maps allow one- and few-step generation.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026🧠 AI

This paper shows that training-free relaxed (lossy) verification in speculative decoding causes length inflation: completions get longer, mostly through extended reasoning traces, which offsets gains in accepted tokens per round. Across twelve target-drafter pairs, relaxed rules accept 1.34x as many tokens per round but produce 1.54x longer completions. Rules that cut verifier rounds usually lose substantial accuracy, so the authors argue completion length and verifier rounds must be reported.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026🧠 AI

DLite is a history-free speculative decoding framework for agentic LLM workloads. It reuses the target model's contextual hidden states and predicts locally without keeping the draft model's historical KV cache, which avoids the up to 80% acceptance-length drop seen with long, multi-turn contexts. A lightweight sequential head restores intra-block causal dependencies, and a two-stage training strategy is used. It reaches 2-6 acceptance length and beats DSpark by up to 70%.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026🧠 AI

Announces Reflection's Beam-1.0-501B-A23B, an Apache 2.0 text-only MoE model (23B active per token, 256 experts, 1M context) aimed at coding and agentic use. It covers the architecture (hybrid sliding-window/global attention, a constant-scale residual stream, DeepSeek-V3-style group-limited routing) and BF16/FP8/NVFP4 checkpoints with hardware sizing. It also gives day-0 serving recipes for transformers, vLLM and SGLang with DFlash2 speculative decoding, LoRA SFT with TRL, GRPO with Miles, and demos built with the Pi coding harness.

huggingface.coarticlePublished and added 16 Oct 2026🧠 AI

SP$^2$ec (Single-Peak SPeculative decoding) is a training-free bandit algorithm that adapts the draft configuration of self-speculative decoding per prompt for vision-language models. It relies on the single-peak throughput pattern seen in SWIFT, KnapSpec, SpecVLM and the authors' cosSim method. It minimizes a new wall-time regret measure and has theoretical guarantees of near-optimal configuration with negligible exploration cost. On Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B it improves over the original self-speculative methods by up to 11% on visual and textual tasks.

arxiv.orgpaperAdded 14 Oct 2026, published 13 Oct 2026🧠 AI

ReplaySSM represents state-space model state as a periodically updated checkpoint plus a buffer of recent SSM inputs, separating output computation from checkpoint updates. This avoids state writes on most decoding steps (roughly halving dominant state memory traffic), makes speculative-decoding rollback a buffer-pointer operation, and lets a low-precision checkpoint serve outputs while a higher-precision one drives recurrent updates. Implemented in vLLM for Mamba-2 and Gated DeltaNet, it gives 1.20–1.47x decoding speedup at batch size 256 across four hybrid models and about 1.8x the throughput of vLLM speculative decoding on two 120B-class models.

arxiv.orgpaperAdded 12 Oct 2026, published 9 Oct 2026🧠 AI

Adaptive DHiSparse extends HiSparse (host-memory KV storage that fetches only selected entries) to draft-model speculative decoding methods DSpark and DFlash2. It protects the required working set, keeps extra history while GPU memory is spare, and recovers the rest by exact fused retrieval. Queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 reaches 793 output tokens/s per decode GPU (1.45x HiSparse). With the same latency bounds it sustains up to 4x the concurrency. On DeepSeek-V4.1-Flash at 500K context, it delivers 1.85x the output of resident serving with DSpark.

arxiv.orgpaperAdded 12 Oct 2026, published 9 Oct 2026🧠 AI

The paper formalizes speculative decoding as a Markov reward process and derives the Expected Decoding Rounds (EDR) objective, a hyperparameter-free training loss equal to the expected number of decoding rounds, addressing cross-round coupling that prior block-local surrogate objectives ignore. It derives an exact temporal-difference gradient for unbiased stochastic optimization and an offline evaluator for comparing drafters on shared rollouts without running speculative decoding. Finetuning the DSpark and DFly drafters with EDR improves mean accepted length and beats prior training objectives across nine benchmarks in math, code, and chat.

arxiv.orgpaperAdded 8 Oct 2026, published 7 Oct 2026🧠 AI

This arXiv paper presents the first systematic study of security implications of speculative decoding, a technique that speeds up LLM inference using a smaller draft model to propose tokens verified by the target model. The authors find a security-utility asymmetry: lossy speculative decoding methods raise jailbreak and prompt-injection attack success rates much faster than they degrade utility, with security degradation traced to early tokens generated by the draft model. They propose SecureSD, which applies stricter verification to early-position draft tokens, improving security while preserving inference efficiency and utility on security and utility benchmarks.

arxiv.orgpaperAdded 7 Oct 2026, published 6 Oct 2026🛡️ Security

The paper proposes Entropy-Aware Mixing, a knowledge-distillation method where a small student model's per-token distribution is dynamically interpolated with a stronger teacher model's distribution, gated by the student's own predictive entropy, to avoid "confident conflicts" where the student must imitate low-probability continuations despite being confident in a different one. The authors implement both convex and geometric interpolations under two training regimes — offline trace generation via speculative decoding followed by SFT, and on-policy forward-KL distillation — and show it stabilises distillation, improving in- and out-of-distribution math reasoning while better preserving general capabilities than fixed-teacher supervision, with offline traces favouring concave entropy schedules and on-policy training favouring linear/convex ones.

arxiv.orgpaperAdded 6 Oct 2026, published 5 Oct 2026🧠 AI

Daily AINews digest (Latent Space) covering an October 2026 news cycle: OpenAI's cheaper/faster GPT-6.1 Sol and Anthropic's Claude Sonnet 5.5 reshaping Agent Arena's cost-performance frontier, a wave of decision-model releases (Cloudflare Clef, llama.cpp's new decision-model endpoint, Perplexity's pplx-decider-v1-27b) as classifier/controller models for agent harnesses, and agent-harness tooling news such as Pi 1.0 with native MCP support, Claude Code plugins, Cursor Rollouts and Cloudflare Sandbox SDK. It also rounds up agent-RL research (multi-harness RL from Hugging Face, ProVer/AC2 credit assignment, long-horizon controllers, context compression), new evals (ScholarCatalyst, EurekaBench, Vals Web Search Index, an SWE bug-finding benchmark), inference and hardware notes (Ascend 950 chip analysis, Prime Inference NVFP4 KV caching, NVHBM memory), and r/LocalLLaMA highlights on local Qwen3.8-27B inference and MTP speculative decoding in llama.cpp.

latent.spacearticlePublished and added 3 Oct 2026🧠 AI

Hy4 preview is Tencent's new flagship Mixture-of-Experts language model: 770B total parameters with 49B activated per token across 78 layers (256 routed experts + 1 shared expert, top-8 routing), plus a native 10B multi-token-prediction layer for speculative decoding. It introduces Gated DeepSeek Sparse Attention with IndexCache for cross-layer index reuse and identity Hyper-Connections (iHC) on the residual path, and in blind internal evaluations on engineering tasks it edges out GLM-5.3 and Kimi K3. Weights (including an FP8 variant) are released under Apache 2.0 on Hugging Face with vLLM/SGLang deployment recipes and AngelSlim-based quantization/compression tooling.

huggingface.corepoAdded 2 Oct 2026🧠 AI

EdgeAgent is a cross-layer LLM inference system for running multi-agent workloads on edge devices with CPU-GPU unified memory architectures such as Apple M-series SoCs. It introduces zero-copy UMA-aware tensor parallelism to saturate both CPU and GPU, a dynamic speculative-decoding draft-budget scheduler adapting to sequence predictability, and an asynchronous suspend-and-yield mechanism that evicts stalled agents during tool-call latencies. On an Apple M4 SoC it reports 1.29x speedup from UMA-aware execution alone and 1.77x with full agent-aware scheduling, versus batched speculative decoding.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026🧠 AI

An arXiv preprint that empirically tests speculative decoding on consumer hardware (Intel Arc B570, NVIDIA RTX, Apple M4-class systems), distinguishing between 'execution-path qualification' (whether a speculative predictor's output matches the target model's exact generation trajectory) and the realized latency/cost of running speculation. The authors find that a target-only, draft-free predictor matches greedy output on most repeat-stable prompts but does not guarantee qualification or identify a unique cause of divergence, and that newly qualified RTX serialization and other configurations (358H, M4 Guard) are often slower than target-only decoding once real costs are counted. The paper's conclusion is that these results are diagnostic engineering findings specific to the tested hardware/software paths, not evidence of a portable decoding speedup.

arxiv.orgpaperAdded 5 Oct 2026, published 2 Oct 2026🧠 AI

DRelay is a speculative decoding technique for LLM inference that repairs parallel draft-token selections before target-model verification, using a global reader over the entire draft block plus a causal selector to decide whether to keep or replace each candidate token, extending the accepted prefix length. Trained jointly with the draft backbone via a combined candidate-support and position-weighted repair loss, it is evaluated on eight benchmarks on an H800 GPU and, under SGLang serving, improves end-to-end decoding speedup substantially over DFlash, Domino, and DSpark baselines.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026🧠 AI

AgSpec is a retrieval-based speculative decoding framework built for coding-agent pipelines. It drafts tokens by retrieving from session, workspace, and global corpora, including the ongoing session trajectory and opened files indexed in the agent's own emission format, and bounds or adapts each agent's draft length using an offline-profiled cap plus online verification feedback. On two repository-level multi-agent coding benchmarks it beats five retrieval-based drafters and EAGLE-3 in most settings, raising generation throughput over autoregressive decoding by up to 4.37x at batch size 1 and 4.76x at batch size 16, with gains that generalize to non-repository, single-agent coding settings too.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026🧠 AI

DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters, 1M-token context, and a novel Causal Encoder-Decoder (CED) architecture where the decoder's global KV cache is projected from encoder hidden states rather than recomputed per layer, activating only 8B/16B parameters per token during prefill/decode. It introduces Compressed Sparse Attention 2 (CSA2) with FP4 KV caching to cut the global KV cache footprint to 890 bytes/token (~1/4 of DeepSeek-V4-Flash, ~1/437 of DeepSeek-V1), alongside Engram conditional memory, DSpark speculative decoding, and a continuously controllable reasoning-effort setting (1–100). The card documents pre/post-training recipes (45T tokens, SFT→RL→on-policy distillation with synthetic agent environments) and reports results across agentic and code benchmarks (Terminal-Bench 2.1/3.0/4.0, DeepSWE v1.1, GPQA Diamond, Chartography, ZeroBench, AutomationBench), plus deployment instructions for Transformers, vLLM, SGLang, and Docker.

huggingface.corepoAdded 30 Sep 2026🧠 AI

Release notes for vLLM v0.30.0, a major update to the open-source LLM inference/serving engine spanning 762 commits. Highlights include new model support (DeepSeek-V4.1-Flash, GLM-5.3-Flash, Kimi K3, Qwen3.8-Flash-Next, K2-Horizon, Cohere Compass and others), a "Fast Start" GPU weight-cache daemon for near-instant engine restarts via CUDA IPC, Gumbel-max watermarking for generated text, a HiSparse host-memory tier for sparse-MLA KV offload, Model Runner V2 improvements (dual-batch overlap, speculative decoding under pipeline parallelism), expanded quantization support (NVFP4, W4A16 DSA, AutoRound), and hardware-specific tuning across NVIDIA, AMD ROCm, Intel XPU and CPU backends.

github.comrepoAdded 28 Sep 2026🧠 AI

ASPIRE is a non-synchronized batched self-speculative decoding framework for long-context LLM inference, where requests independently choose when to verify instead of sharing a global draft-verify schedule. It combines a unified mixed forward pass, a per-request online speculation scheduler using acceptance-rate estimates and a batch-aware cost model, and an intra-draft refresh layer that periodically applies full attention to reduce staleness. Across three models and five reasoning/long-context benchmarks it reports 1.70-4.58x decoding throughput speedup over autoregressive baselines, ~27% better than prior self-speculative baselines.

arxiv.orgpaperAdded 26 Sep 2026🧠 AI

Speculative Uncertainty (SU) repurposes speculative decoding in reverse: a small open-weight draft model scores an agent's already-generated trajectory in a single forward pass, producing a failure-likelihood signal from output tokens alone without needing logits, weights, or repeated sampling. Phase-aware features separate reasoning and action spans and are calibrated against a verifiable objective to drive a pre-execution veto gate. On software engineering agents built on Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, the gate cuts execution error rate by 6-8 percentage points and token cost by 14-19%, transferring to out-of-distribution benchmarks without retraining.

arxiv.orgpaperAdded 26 Sep 2026🧠 AI

Liquid AI releases LFM2.5-VL-3B-DSpark, a 280M-parameter attention-only draft model (4 layers, block size 9, +8.9% parameters) for speculative decoding of the LFM2.5-VL-3B vision-language model. It reports decode speedups of 2.30–3.13x on MLX (M5 Max), 1.57–2.14x on llama.cpp (M3 Ultra) and up to 2.66x on H100, with end-to-end gains of 1.30–2.62x, evaluated on six MMSpec vision tasks. It explains why prefill and vision encoding limit end-to-end gains (Amdahl's law) and gives launch commands for SGLang, llama.cpp and MLX-VLM.

huggingface.coarticleAdded 26 Sep 2026, published 24 Sep 2026🧠 AI

A structured guide walking through the layered techniques used to make LLM inference faster and cheaper: understanding the compute-bound prefill vs. memory-bandwidth-bound decode phases, KV caching with PagedAttention (as implemented in vLLM) and prefix caching, static/dynamic/continuous batching, attention optimizations (MQA, GQA, FlashAttention), model compression (GPTQ/AWQ quantization, 2:4 structured sparsity, knowledge distillation), speculative decoding for latency-critical single-request scenarios, and scaling via tensor/pipeline parallelism and prefill-decode disaggregation. It ties each technique to the specific bottleneck it addresses, making it a practical map for choosing optimizations based on latency vs. throughput requirements.

machinelearningmastery.comarticleAdded 28 Sep 2026, published 21 Sep 2026🧠 AI

Pooled runs one open model (Qwen 3.6 35B MoE, a 27B dense hybrid, smaller Qwen3/SmolLM) across devices in a browser "room". Each tab holds a slice of layers and runs them on its own GPU with custom WGSL kernels. Hidden states pass over WebRTC, and speculative decoding is checked against plain decoding by golden tests. It also ships a Code-mode agent loop with a sandboxed preview, and an npx CLI bridge (@pooled/cli) that exposes the room as an OpenAI Chat Completions/Responses and Anthropic Messages endpoint with grammar-constrained tool calling, so Codex CLI, Claude Code and opencode can use it. The README includes latency-injected benchmarks, a comparison with exo, Petals, llama.cpp RPC and WebLLM, and a threat model.

github.comrepoAdded 16 Oct 2026, published 1 Sep 2026🧠 AI

A 10-week, 30-minutes-a-day roadmap (GitHub repo "time-to-first-token") that walks engineers through building one continuously-evolving OpenAI-compatible inference service rather than disconnected tutorials: standing up vLLM and SGLang, reading the PagedAttention/RadixAttention source, instrumenting Grafana dashboards for TTFT/inter-token latency/throughput/queue depth, load-testing past 1000 concurrent requests with GuideLLM, benchmarking FP8/INT4/AWQ quantization and speculative decoding, configuring KV cache eviction, deploying prefill/decode disaggregation on Kubernetes, and building a cost/latency/quality router. It closes with a checklist of common public-benchmark mistakes (sweep request rate, report p50/p95/p99, separate TTFT from inter-token latency, pin versions) and pointers to Stanford CS336, MIT EfficientML.ai, vLLM Office Hours, and GPU MODE for further study.

github.comrepoAdded 2 Oct 2026, published 2 Aug 2026🧠 AI