⚡ Inference & Serving207

Project Suncatcher is a Google research effort to power AI compute in space. Its prototype satellite launched on 1 October 2026 and, after commissioning, returned the first AI response from a data-center-class TPU in orbit: a haiku from the onboard Gemma 4 open model, downlinked over Svalbard. Next come thermal-stress and radiation tests, then twin interconnected satellites in 2027.

blog.googlearticlePublished and added 30 Oct 2026

Doomformer compiles integer neural networks into ZDoom maps that run in ViZDoom. Voodoo dolls do the arithmetic, and sector light levels and plane heights hold the values. Python reads every value back and compares it with a NumPy integer reference at tolerance 0. It runs MNIST (97.16 % on all 10,000 digits, every pass exact), small transformers, a learning perceptron and a SUBLEQ machine. It is about 23,000 times slower than NumPy on a small MLP.

github.comrepoAdded 30 Oct 2026, published 29 Oct 2026

TextSeal2 is a distortion-free watermark for LLM output that keeps its signal under speculative decoding. It merges two or more keys into one virtual key coupled to fresh noise through AutoMark's Gaussian copula, and verifies drafts with an extra exponential race. On three LLMs it detects 8 to 22 points more 100-token answers than TextSeal at a 10^-3 false-positive rate.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

This paper derives bounds on predicting an LLM's decode length from prefill-phase hidden states, a primitive for inference scheduling. It models decoding as a stopped stochastic process, giving per-prompt normality and an affine variance law σ=αμ+b. Predictors (MLP, EGTP, FiLM, soft-label, Mixture-of-Experts) converge near the bound, so system error dominates model error. It proposes fixing model, data and sampling in benchmarks and reporting relative-error metrics.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

This preprint proposes a unified post-training quantization framework for large language models. It derives a Hessian-guided approximation of the output change caused by joint weight and activation quantization, using a tractable Fisher approximation. Quantization ranges, linear transformations and weight rounding are optimized together under one output-sensitivity objective. The authors report gains over state-of-the-art transformation and rounding methods in channel-wise and block-wise settings.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

MAServeBench is a benchmark that treats multi-agent collaboration as a configurable dimension of LLM serving workloads. Its composable stages model how agents coordinate and exchange context in documented multi-agent workflows, and Hierarchical Semantic Traces record operations and dependencies for reproducible comparisons. Task-linked measurements tie workloads to GPU/NPU behavior, showing serving efficiency does not always mean timely task completion. Code is released.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

This paper measures the energy cost of speculative decoding in SGLang with CUDA graphs on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. Savings depend on drafted-token acceptance, which is lower on conversation than code and lower on Romansh than English. Savings shrink as batch size grows, and block drafters can push verification into the compute-bound regime, where speculation raises energy per token.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

SGLang announces Day-0 support for Mistral Large 4 ("Le Chonk"), saying it is the first engine to serve the model on NVIDIA Vera Rubin. The post describes the model as Mistral's largest to date: a natively multimodal 1T-parameter mixture-of-experts model covering instruct, reasoning, coding and agentic workflows. The fetched text is truncated, so benchmark claims and launch commands are not visible.

x.comthreadAdded 30 Oct 2026, published 29 Oct 2026

AnySpeechRecognizer and AnyDecisionModel are two Swift packages from Hugging Face. The first mirrors Apple's SpeechAnalyzer API with swappable engines (NVIDIA Parakeet, Alibaba Qwen3-ASR on Core ML) and adds Sortformer speaker diarization. The second returns typed yes/no, choice and score answers with probabilities via MLX, Core AI, llama.cpp, Jev or Ollama. The post gives M4 Max timings, model sizes and calibration options.

huggingface.coarticlePublished and added 29 Oct 2026

Mistral Large 4 (Le Chonk) is a multimodal, multilingual Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context, released under the Mistral Ecosystem Open-Weight License. The post covers its MLA and DeepSeek Sparse Attention architecture, a 1.6B Pixtral encoder, and benchmarks including 82% on the Artificial Analysis Cyber Index and 93% on Cybench. It gives run recipes for transformers, vLLM, llama.cpp, TRL fine-tuning and Inference Endpoints.

huggingface.coarticlePublished and added 29 Oct 2026

Decrypt reports, citing The Information, that OpenAI plans to buy Google TPUs rather than rent them, to reduce dependence on Nvidia. Neither company has confirmed, and price, volume and timing are undisclosed. The article covers OpenAI's TPU optimization job listing, its Broadcom-built Jalapeño inference chip due end of 2026, Nvidia and AMD deals, and Anthropic's multi-gigawatt TPU agreement.

decrypt.coarticlePublished and added 28 Oct 2026

QbD-VLA is a mixed-precision post-training quantization method for vision-language-action models. It uses task success from short closed-loop evaluations to assign weight and activation bit widths, accounting for interactions between components. On four VLA models across all four LIBERO suites it holds task performance at about 4-bit weights and under 6-bit activations. The gain on SIMPLER is small. On a Franka Research 3 arm, X-VLA shows no detected loss, but π0.5 fails.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

FlatCake is a KV cache storage architecture for LLM serving that treats host DRAM and NVMe SSD as peer media in one 64-bit address space, instead of a DRAM-over-SSD tier. Blocks are placed once, in session-contiguous, layer-major order, and streamed via GPUDirect Storage. On a Claude Code trace with DeepSeek-V4-Flash it holds a 94% hit rate (Mooncake: 64%) and cuts peak-load mean TTFT 1.6x.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

Survival LLM is a free, open-source desktop app for macOS (Apple Silicon) and 64-bit Windows that runs an AI model and an offline emergency library on the user's computer. It checks the machine's memory, downloads a fitting model, and answers from sources such as Wikipedia, WikiMed, Ready.gov and CDC guidance, citing each article. Users can add their own files; version 0.3.2 is dated October 2026.

brangus.iotoolAdded 28 Oct 2026, published 27 Oct 2026

Photon Capital's thread covers the 27 October 2026 NVIDIA and Oracle announcement of general availability of Vera Rubin NVL72 and Vera CPUs on OCI. It lists specs (Spectrum-6 at 102.4Tbps, HGX Rubin NVL8 with Xeon 6, 150,000 initial Vera CPUs) and OpenAI's use of Vera CPUs for agentic sandboxes. The commentary argues CPU capacity can cut cost per task and shorten KV cache residency in GPU memory.

x.comthreadAdded 28 Oct 2026, published 27 Oct 2026

Metacognitive Decoding is a lightweight plug-in controller for diffusion language model backbones. It learns when to finalize tokens and how far generation should extend, and is trained on agent trajectories to balance correctness against generation cost. Across five benchmarks it cuts inference FLOPs by 20.6% on average versus the most efficient baseline, with competitive task accuracy. The abstract names no specific benchmarks or backbones.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

LFM3 is Liquid AI's new family of open-weight multimodal models, six models of about 2B parameters branched from LFM3-1.7B-Base (22T pre-training tokens, 256K context). The hybrid backbone uses 13 gated short-convolution, 5 sliding-window attention and 6 global attention layers. Variants cover text, vision with video, audio, omni and two decision models, with on-device speed numbers, fine-tuning demos and robotics extensions.

liquid.aiarticleAdded 28 Oct 2026, published 27 Oct 2026

CRISP is a combination-aware framework that co-designs pruning and runtime scheduling for Mixture-of-Experts inference on memory-constrained edge platforms. It reorders neurons by importance so one stored expert supports several execution widths, gives hot, warm and cold expert combinations different widths, and overlaps expert loading with computation. On Mixtral-8x7B and DeepSeek-MoE-16B it raises throughput up to 2.1x at the same memory budget, with accuracy loss limited to 3.6%.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

Walkthrough of running OpenCode against Qwen3.5 9B served by Ollama on a Mac, with Wi-Fi switched off. The author fixes a one-line accumulation bug in an expense calculator and confirms all five unittest tests pass. It covers the opencode.json setup, memory budgeting, permission prompts and a pytest-versus-unittest detour. Two further exercises (CSV report, log analyzer) were not yet run offline.

latent.spacewriteupPublished and added 27 Oct 2026

p-DCIM is a 28 nm SRAM digital compute-in-memory macro (0.16 mm2, 0.41 mW) that works as an Ising machine for RAG evidence selection. Selection is cast as a QUBO problem, and the same MAC datapath computes embedding similarities and annealing steps. Replacing LLM reranking with it gives a selection-energy estimate over six orders of magnitude lower and cuts total energy per query by 67%, with comparable token-level F1 on the evaluated examples.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

DCKV is a differentiable codec for the prompt KV cache of hybrid language models, which keep a KV cache only in their few full-attention layers. Transform, quantizer and entropy models (conditioned on prompt tokens and already decoded neighbouring positions) are trained end to end against a frozen model. On Qwen3.5-2B it stores a token in over 55 times fewer bytes than bf16 with at most 1.8 points lost on three long-context benchmarks, and over 100 times fewer with at most 3.6 points lost.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

AxisQ is a post-training quantization framework for camouflaged object detection (COD) that allocates activation bits by feature depth. It uses the channel domain for shallow features and the local-frequency domain for deep features, under a fixed per-layer average bit budget. It adds a boundary-aware Hessian approximation for reconstruction. On CAMO at W2A2, weighted F-measure improves 12.7% for CFRN and 33.3% for CamoFormer-P over the strongest baselines.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

AINews issue from Latent Space on 27 October 2026, led by Anthropic's S-1 filing for a Nasdaq listing (ticker ANT). It reports H1 2026 revenue of $16.3B, an 85% gross margin, $974M Q2 operating profit and 48% of revenue through cloud marketplaces, with reactions from named analysts. It also covers model releases, inference economics and agent tooling. The page text is truncated.

latent.spacearticlePublished and added 27 Oct 2026

OblivDGL is a framework for secure GNN training and inference in machine-learning-as-a-service settings, modelled on Deep Graph Library. It uses an oblivious message passing paradigm and a new oblivious graph representation built on function secret sharing, revealing only the total node and edge counts. On four benchmark graphs it cuts training communication up to 1024x with up to 14.3x speedup, and inference communication up to 754x with 4.19x speedup.

eprint.iacr.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

The paper folds the per-key logit bias of Attention Matching (AM) KV cache compaction into the keys along one closed-form direction per head, fitted on context queries. The result is a plain cache that FlashAttention and vLLM paged caches can use, with output error bounded by how far each query's projection on that direction is from 1. Unmodified vLLM serves folded caches through a KV connector. On QuALITY and LongHealth the fold is within noise of AM in 13 of 14 settings. Under AM's published per-head budgets it is within noise in 9 of 10.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

Cloudflare's postmortem of a tenant isolation bug in the GLM-5.3 model on Workers AI, active 27 August to 9 September 2026. Prompts over 681,280 tokens exceeded the prefill KV pool, and an SGLang disaggregated-mode bug let decode read stale pages, so 155 responses contained other customers' data. It covers the timeline, config and engine fixes (SGLang v0.5.22), and recommendations for operators.

blog.cloudflare.comwriteupAdded 27 Oct 2026, published 26 Oct 2026

Ollama overview thread by Akash Rajagopal on X. It lists six steps: install on macOS, Linux, Windows or Docker; run `ollama run gemma4`; use the REST API on localhost:11434 (/api/chat); call it from ollama-python and ollama-js; llama.cpp as the README's supported backend; and `ollama launch claude` to connect Claude Code. Codex, OpenCode, Copilot CLI and Droid also work. The thread cites 184k GitHub stars and an MIT licence.

x.comthreadAdded 27 Oct 2026, published 26 Oct 2026

Hob is an MIT-licensed desktop app from Frigade that runs open-weight models such as Gemma and gpt-oss locally on Mac, Windows and Linux. It picks the best open-weight model that fits the user's computer and sets it up. It is positioned as a free, private, offline alternative to Claude and ChatGPT for tasks like filling out PDFs and working through documents. The page gives no technical detail on its runtime.

github.comrepoAdded 27 Oct 2026, published 26 Oct 2026

Ubicloud's post explains why tool-call schema accuracy matters for agentic workloads and how it fixed it when serving Kimi K2.6 and DeepSeek V4 Flash. With vLLM 0.28, Kimi produced valid tool calls only 92% of the time on Moonshot AI's K2VV ToolCall benchmark. Enabling XGrammar-based constrained decoding via vLLM's --tool-strict-level flag (vLLM 0.31) raised schema accuracy to 100%.

ubicloud.comwriteupAdded 27 Oct 2026, published 26 Oct 2026

SemiAnalysis claims AMD chips running vLLM can earn up to $3 per chip-hour in profit serving the open-source Kimi K3 model, even after higher chip rental prices following Claude Code's rise. Kimi K3 has over 2.8 trillion parameters and fits AMD's 288GB per chip, while B200 cannot hold it in one server. B200 must then use tensor or pipeline parallelism over 400Gbit/s InfiniBand or Ethernet, which SemiAnalysis says is 18x slower than NVLink. The post gives no methodology.

x.comthreadAdded 26 Oct 2026, published 25 Oct 2026