Skip to content

⚡ Inference & Serving207

ListGrid
What you seeeverything
RecentOldestAddedA–Z
All types207Papers80Articles69Repos25Threads17
Write-ups8Talks3Tools3Courses2
⚡ Inference & Serving
blog.googlearticle

We’ve received our first response from the Project Suncatcher prototype satellite.

Project Suncatcher is a Google research effort to power AI compute in space. Its prototype satellite launched on 1 October 2026 and, after commissioning, returned the first AI response from a data-center-class TPU in orbit: a haiku from the onboard Gemma 4 open model, downlinked over Svalbard. Next come thermal-stress and radiation tests, then twin interconnected satellites in 2027.

TPUGemma
GoogleGemma 4Project Suncatcher+1

inference, ai-infra· Published and added 30 Oct 2026

github.comrepo

Show HN: I Turned DOOM into a Transformer

Doomformer compiles integer neural networks into ZDoom maps that run in ViZDoom. Voodoo dolls do the arithmetic, and sector light levels and plane heights hold the values. Python reads every value back and compares it with a NumPy integer reference at tolerance 0. It runs MNIST (97.16 % on all 10,000 digits, every pass exact), small transformers, a learning perceptron and a SUBLEQ machine. It is about 23,000 times slower than NumPy on a small MLP.

ViZDoomZDoomACS+10
DoomformerDOOMFreedoom+6

research, inference· Added 30 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

TextSeal2: Distortion-Free LLM Watermarking with Copula Mixing and Memoryless Races

TextSeal2 is a distortion-free watermark for LLM output that keeps its signal under speculative decoding. It merges two or more keys into one virtual key coupled to fresh noise through AutoMark's Gaussian copula, and verifies drafts with an extra exponential race. On three LLMs it detects 8 to 22 points more 100-token answers than TextSeal at a 10^-3 false-positive rate.

LLMspeculative decodingGumbel-max watermarking+1
TextSeal2TextSealAutoMark+2

research, inference· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

On the Fundamental Limits of Prefill-Based Decoding Length Prediction

This paper derives bounds on predicting an LLM's decode length from prefill-phase hidden states, a primitive for inference scheduling. It models decoding as a stopped stochastic process, giving per-prompt normality and an affine variance law σ=αμ+b. Predictors (MLP, EGTP, FiLM, soft-label, Mixture-of-Experts) converge near the bound, so system error dominates model error. It proposes fixing model, data and sampling in benchmarks and reporting relative-error metrics.

LLMMixture of ExpertsMLP+2
arXiv

research, inference, evals· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

A Unified Hessian-Guided Framework for Post-Training Quantization

This preprint proposes a unified post-training quantization framework for large language models. It derives a Hessian-guided approximation of the output change caused by joint weight and activation quantization, using a tractable Fisher approximation. Quantization ranges, linear transformations and weight rounding are optimized together under one output-sensitivity objective. The authors report gains over state-of-the-art transformation and rounding methods in channel-wise and block-wise settings.

LLMpost-training quantizationHessian-guided approximation+1
arXiv

research, inference· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads

MAServeBench is a benchmark that treats multi-agent collaboration as a configurable dimension of LLM serving workloads. Its composable stages model how agents coordinate and exchange context in documented multi-agent workflows, and Hierarchical Semantic Traces record operations and dependencies for reproducible comparisons. Task-linked measurements tie workloads to GPU/NPU behavior, showing serving efficiency does not always mean timely task completion. Code is released.

LLM
MAServeBencharXiv

evals, inference, agents· Added 31 Oct 2026, published 29 Oct 2026

arxiv.orgpaper

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

This paper measures the energy cost of speculative decoding in SGLang with CUDA graphs on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. Savings depend on drafted-token acceptance, which is lower on conversation than code and lower on Romansh than English. Savings shrink as batch size grows, and block drafters can push verification into the compute-bound regime, where speculation raises energy per token.

SGLangCUDALLM+4
NVIDIAarXivRomansh

inference, research· Added 31 Oct 2026, published 29 Oct 2026

x.comthread

SGLang (@sgl_project) on X

SGLang announces Day-0 support for Mistral Large 4 ("Le Chonk"), saying it is the first engine to serve the model on NVIDIA Vera Rubin. The post describes the model as Mistral's largest to date: a natively multimodal 1T-parameter mixture-of-experts model covering instruct, reasoning, coding and agentic workflows. The fetched text is truncated, so benchmark claims and launch commands are not visible.

SGLangMixture of Experts
MistralMistral Large 4NVIDIA

inference, models· Added 30 Oct 2026, published 29 Oct 2026

huggingface.coarticle

Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms

AnySpeechRecognizer and AnyDecisionModel are two Swift packages from Hugging Face. The first mirrors Apple's SpeechAnalyzer API with swappable engines (NVIDIA Parakeet, Alibaba Qwen3-ASR on Core ML) and adds Sortformer speaker diarization. The second returns typed yes/no, choice and score answers with probabilities via MLX, Core AI, llama.cpp, Jev or Ollama. The post gives M4 Max timings, model sizes and calibration options.

SwiftCore MLMLX+10
AnySpeechRecognizerAnyDecisionModelAnyLanguageModel+17

multimodal, inference, models· Published and added 29 Oct 2026

huggingface.coarticle

Welcome Le Chonk: Mistral Large 4 on Hugging Face

Mistral Large 4 (Le Chonk) is a multimodal, multilingual Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context, released under the Mistral Ecosystem Open-Weight License. The post covers its MLA and DeepSeek Sparse Attention architecture, a 1.6B Pixtral encoder, and benchmarks including 82% on the Artificial Analysis Cyber Index and 93% on Cybench. It gives run recipes for transformers, vLLM, llama.cpp, TRL fine-tuning and Inference Endpoints.

Mixture of ExpertsTransformersvLLM+11
Mistral Large 4MistralArtificial Analysis+15

models, inference, evals· Published and added 29 Oct 2026

decrypt.coarticle

OpenAI Plans to Buy Google's AI Chips as It Looks Beyond Nvidia: Report

Decrypt reports, citing The Information, that OpenAI plans to buy Google TPUs rather than rent them, to reduce dependence on Nvidia. Neither company has confirmed, and price, volume and timing are undisclosed. The article covers OpenAI's TPU optimization job listing, its Broadcom-built Jalapeño inference chip due end of 2026, Nvidia and AMD deals, and Anthropic's multi-gigawatt TPU agreement.

TPUJalapeñoGoogle Cloud
OpenAIGoogleNVIDIA+10

inference, ai-infra· Published and added 28 Oct 2026

arxiv.orgpaper

Quantize by Doing: Task-Guided Mixed-Precision Quantization of Vision-Language-Action Models

QbD-VLA is a mixed-precision post-training quantization method for vision-language-action models. It uses task success from short closed-loop evaluations to assign weight and activation bit widths, accounting for interactions between components. On four VLA models across all four LIBERO suites it holds task performance at about 4-bit weights and under 6-bit activations. The gain on SIMPLER is small. On a Franka Research 3 arm, X-VLA shows no detected loss, but π0.5 fails.

vision-language-action model
QbD-VLALIBEROSIMPLER+4

research, inference, multimodal· Added 30 Oct 2026, published 28 Oct 2026

arxiv.orgpaper

FlatCake: A Flat KVCache Storage Architecture for LLM Serving

FlatCake is a KV cache storage architecture for LLM serving that treats host DRAM and NVMe SSD as peer media in one 64-bit address space, instead of a DRAM-over-SSD tier. Blocks are placed once, in session-contiguous, layer-major order, and streamed via GPUDirect Storage. On a Claude Code trace with DeepSeek-V4-Flash it holds a 94% hit rate (Mooncake: 64%) and cuts peak-load mean TTFT 1.6x.

GPUDirect StorageNVMe SSDLLM+3
FlatCakeMooncakeMooncake-GDS+3

inference, research· Added 30 Oct 2026, published 28 Oct 2026

brangus.iotool

Survival LLM: AI assistant and emergency library that runs on your own computer

Survival LLM is a free, open-source desktop app for macOS (Apple Silicon) and 64-bit Windows that runs an AI model and an offline emergency library on the user's computer. It checks the machine's memory, downloads a fitting model, and answers from sources such as Wikipedia, WikiMed, Ready.gov and CDC guidance, citing each article. Users can add their own files; version 0.3.2 is dated October 2026.

LLMRAGmacOS+1
Survival LLMWikipediaWikiMed+9

llm-apps, inference· Added 28 Oct 2026, published 27 Oct 2026

x.comthread

Photon Capital (@PhotonCap) on X

Photon Capital's thread covers the 27 October 2026 NVIDIA and Oracle announcement of general availability of Vera Rubin NVL72 and Vera CPUs on OCI. It lists specs (Spectrum-6 at 102.4Tbps, HGX Rubin NVL8 with Xeon 6, 150,000 initial Vera CPUs) and OpenAI's use of Vera CPUs for agentic sandboxes. The commentary argues CPU capacity can cut cost per task and shorten KV cache residency in GPU memory.

Oracle Cloud InfrastructureOCI Kubernetes EngineKubernetes+6
NVIDIAOracleOpenAI+7

inference, ai-infra, agents· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

Metacognitive Decoding: Efficient Inference for Diffusion Language Agents

Metacognitive Decoding is a lightweight plug-in controller for diffusion language model backbones. It learns when to finalize tokens and how far generation should extend, and is trained on agent trajectories to balance correctness against generation cost. Across five benchmarks it cuts inference FLOPs by 20.6% on average versus the most efficient baseline, with competitive task accuracy. The abstract names no specific benchmarks or backbones.

diffusion model
arXiv

research, agents, inference· Added 28 Oct 2026, published 27 Oct 2026

liquid.aiarticle

Introducing LFM3: Every Modality for Specialized Intelligence

LFM3 is Liquid AI's new family of open-weight multimodal models, six models of about 2B parameters branched from LFM3-1.7B-Base (22T pre-training tokens, 256K context). The hybrid backbone uses 13 gated short-convolution, 5 sliding-window attention and 6 global attention layers. Variants cover text, vision with video, audio, omni and two decision models, with on-device speed numbers, fine-tuning demos and robotics extensions.

llama.cppMLXvLLM+17
Liquid AILFM3LFM3-1.7B+32

models, multimodal, inference, training· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

CRISP is a combination-aware framework that co-designs pruning and runtime scheduling for Mixture-of-Experts inference on memory-constrained edge platforms. It reorders neurons by importance so one stored expert supports several execution widths, gives hot, warm and cold expert combinations different widths, and overlaps expert loading with computation. On Mixtral-8x7B and DeepSeek-MoE-16B it raises throughput up to 2.1x at the same memory budget, with accuracy loss limited to 3.6%.

Mixture of Experts
Mixtral-8x7BDeepSeek-MoE-16BarXiv

inference, research· Added 28 Oct 2026, published 27 Oct 2026

latent.spacewriteup

Coding From the Himalayas With an Offline AI Agent

Walkthrough of running OpenCode against Qwen3.5 9B served by Ollama on a Mac, with Wi-Fi switched off. The author fixes a one-line accumulation bug in an expense calculator and confirms all five unittest tests pass. It covers the opencode.json setup, memory budgeting, permission prompts and a pytest-versus-unittest detour. Two further exercises (CSV report, log analyzer) were not yet run offline.

OpenCodeOllamaQwen3.5+4
QwenThomson ReutersLatent Space+4

coding-agents, inference· Published and added 27 Oct 2026

arxiv.orgpaper

p-DCIM: A 28nm Probabilistic SRAM DCIM based Ising Machine for Near-Memory Context Selection in LLM-RAG Pipeline

p-DCIM is a 28 nm SRAM digital compute-in-memory macro (0.16 mm2, 0.41 mW) that works as an Ising machine for RAG evidence selection. Selection is cast as a QUBO problem, and the same MAC datapath computes embedding similarities and annealing steps. Replacing LLM reranking with it gives a selection-energy estimate over six orders of magnitude lower and cuts total energy per query by 67%, with comparable token-level F1 on the evaluated examples.

RAGLLMSRAM+2
arXiv

research, inference, llm-apps· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models

DCKV is a differentiable codec for the prompt KV cache of hybrid language models, which keep a KV cache only in their few full-attention layers. Transform, quantizer and entropy models (conditioned on prompt tokens and already decoded neighbouring positions) are trained end to end against a frozen model. On Qwen3.5-2B it stores a token in over 55 times fewer bytes than bf16 with at most 1.8 points lost on three long-context benchmarks, and over 100 times fewer with at most 3.6 points lost.

Qwen3.5TransformerLLM
DCKVarXiv

research, inference· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

AxisQ: Depth-Adaptive Channel--Frequency Quantization for Camouflaged Object Detection

AxisQ is a post-training quantization framework for camouflaged object detection (COD) that allocates activation bits by feature depth. It uses the channel domain for shallow features and the local-frequency domain for deep features, under a fixed per-layer average bit budget. It adds a boundary-aware Hessian approximation for reconstruction. On CAMO at W2A2, weighted F-measure improves 12.7% for CFRN and 33.3% for CamoFormer-P over the strongest baselines.

CNNTransformer
AxisQCFRNCamoFormer-P+1

research, inference, multimodal· Added 28 Oct 2026, published 27 Oct 2026

latent.spacearticle

[AINews] Anthropic files for $2T IPO in 2 weeks: 1400% growth rate, 48% of revenue through cloud partners, 85% margin, $1B op profit in Q2, $134B cash in bank and $10B lifetime burn

AINews issue from Latent Space on 27 October 2026, led by Anthropic's S-1 filing for a Nasdaq listing (ticker ANT). It reports H1 2026 revenue of $16.3B, an 85% gross margin, $974M Q2 operating profit and 48% of revenue through cloud marketplaces, with reactions from named analysts. It also covers model releases, inference economics and agent tooling. The page text is truncated.

ClaudeClaude CodeHugging Face+7
AnthropicLatent SpaceSemiAnalysis+28

ai-policy, models, inference· Published and added 27 Oct 2026

eprint.iacr.orgpaper

OblivDGL: Oblivious Deep Graph Learning Made Practical

OblivDGL is a framework for secure GNN training and inference in machine-learning-as-a-service settings, modelled on Deep Graph Library. It uses an oblivious message passing paradigm and a new oblivious graph representation built on function secret sharing, revealing only the total node and edge counts. On four benchmark graphs it cuts training communication up to 1024x with up to 14.3x speedup, and inference communication up to 754x with 4.19x speedup.

Deep Graph Librarygraph neural networkfunction secret sharing
OblivDGLNDSS

research, training, inference· Added 28 Oct 2026, published 27 Oct 2026

arxiv.orgpaper

Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels

The paper folds the per-key logit bias of Attention Matching (AM) KV cache compaction into the keys along one closed-form direction per head, fitted on context queries. The result is a plain cache that FlashAttention and vLLM paged caches can use, with output error bounded by how far each query's projection on that direction is from 1. Unmodified vLLM serves folded caches through a KV connector. On QuALITY and LongHealth the fold is within noise of AM in 13 of 14 settings. Under AM's published per-head budgets it is within noise in 9 of 10.

vLLMFlashAttentionHugging Face+3
Attention MatchingQuALITYLongHealth+2

research, inference· Added 28 Oct 2026, published 27 Oct 2026

blog.cloudflare.comwriteup

Cross-request KV cache exposure in the GLM-5.3 Workers AI model (155 responses)

Cloudflare's postmortem of a tenant isolation bug in the GLM-5.3 model on Workers AI, active 27 August to 9 September 2026. Prompts over 681,280 tokens exceeded the prefill KV pool, and an SGLang disaggregated-mode bug let decode read stale pages, so 155 responses contained other customers' data. It covers the timeline, config and engine fixes (SGLang v0.5.22), and recommendations for operators.

SGLangCloudflareGLM+3
Workers AIGLM-5.3GLM-5.2+7

inference, ai-infra· Added 27 Oct 2026, published 26 Oct 2026

x.comthread

Akash Rajagopal (@akashsairaja) on X

Ollama overview thread by Akash Rajagopal on X. It lists six steps: install on macOS, Linux, Windows or Docker; run `ollama run gemma4`; use the REST API on localhost:11434 (/api/chat); call it from ollama-python and ollama-js; llama.cpp as the README's supported backend; and `ollama launch claude` to connect Claude Code. Codex, OpenCode, Copilot CLI and Droid also work. The thread cites 184k GitHub stars and an MIT licence.

Ollamallama.cppClaude Code+10
Gemma 4Akash Rajagopal

inference, coding-agents· Added 27 Oct 2026, published 26 Oct 2026

github.comrepo

Show HN: Hob - Open-source, on device alternative to Claude Cowork and ChatGPT

Hob is an MIT-licensed desktop app from Frigade that runs open-weight models such as Gemma and gpt-oss locally on Mac, Windows and Linux. It picks the best open-weight model that fits the user's computer and sets it up. It is positioned as a free, private, offline alternative to Claude and ChatGPT for tasks like filling out PDFs and working through documents. The page gives no technical detail on its runtime.

GemmaClaudeChatGPT+4
FrigadeHobgpt-oss

inference, llm-apps· Added 27 Oct 2026, published 26 Oct 2026

ubicloud.comwriteup

Fixing Tool-Call Accuracy in vLLM

Ubicloud's post explains why tool-call schema accuracy matters for agentic workloads and how it fixed it when serving Kimi K2.6 and DeepSeek V4 Flash. With vLLM 0.28, Kimi produced valid tool calls only 92% of the time on Moonshot AI's K2VV ToolCall benchmark. Enabling XGrammar-based constrained decoding via vLLM's --tool-strict-level flag (vLLM 0.31) raised schema accuracy to 100%.

vLLMXGrammarOpenCode+2
UbicloudMoonshot AIDeepSeek V4

inference, mcp, evals· Added 27 Oct 2026, published 26 Oct 2026

x.comthread

SemiAnalysis (@SemiAnalysis_) on X

SemiAnalysis claims AMD chips running vLLM can earn up to $3 per chip-hour in profit serving the open-source Kimi K3 model, even after higher chip rental prices following Claude Code's rise. Kimi K3 has over 2.8 trillion parameters and fits AMD's 288GB per chip, while B200 cannot hold it in one server. B200 must then use tensor or pipeline parallelism over 400Gbit/s InfiniBand or Ethernet, which SemiAnalysis says is 18x slower than NVLink. The post gives no methodology.

vLLMClaude CodeNVLink+1
SemiAnalysisAMDKimi K3+1

inference· Added 26 Oct 2026, published 25 Oct 2026

Showing 30 of 207

Show more