⚡ Inference & Serving207

Project Suncatcher is a Google research effort to power AI compute in space. Its prototype satellite launched on 1 October 2026 and, after commissioning, returned the first AI response from a data-center-class TPU in orbit: a haiku from the onboard Gemma 4 open model, downlinked over Svalbard. Next come thermal-stress and radiation tests, then twin interconnected satellites in 2027.

blog.googlearticlePublished and added 30 Oct 2026

Doomformer compiles integer neural networks into ZDoom maps that run in ViZDoom. Voodoo dolls do the arithmetic, and sector light levels and plane heights hold the values. Python reads every value back and compares it with a NumPy integer reference at tolerance 0. It runs MNIST (97.16 % on all 10,000 digits, every pass exact), small transformers, a learning perceptron and a SUBLEQ machine. It is about 23,000 times slower than NumPy on a small MLP.

github.comrepoAdded 30 Oct 2026, published 29 Oct 2026

TextSeal2 is a distortion-free watermark for LLM output that keeps its signal under speculative decoding. It merges two or more keys into one virtual key coupled to fresh noise through AutoMark's Gaussian copula, and verifies drafts with an extra exponential race. On three LLMs it detects 8 to 22 points more 100-token answers than TextSeal at a 10^-3 false-positive rate.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

This paper derives bounds on predicting an LLM's decode length from prefill-phase hidden states, a primitive for inference scheduling. It models decoding as a stopped stochastic process, giving per-prompt normality and an affine variance law σ=αμ+b. Predictors (MLP, EGTP, FiLM, soft-label, Mixture-of-Experts) converge near the bound, so system error dominates model error. It proposes fixing model, data and sampling in benchmarks and reporting relative-error metrics.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

This preprint proposes a unified post-training quantization framework for large language models. It derives a Hessian-guided approximation of the output change caused by joint weight and activation quantization, using a tractable Fisher approximation. Quantization ranges, linear transformations and weight rounding are optimized together under one output-sensitivity objective. The authors report gains over state-of-the-art transformation and rounding methods in channel-wise and block-wise settings.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

MAServeBench is a benchmark that treats multi-agent collaboration as a configurable dimension of LLM serving workloads. Its composable stages model how agents coordinate and exchange context in documented multi-agent workflows, and Hierarchical Semantic Traces record operations and dependencies for reproducible comparisons. Task-linked measurements tie workloads to GPU/NPU behavior, showing serving efficiency does not always mean timely task completion. Code is released.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

This paper measures the energy cost of speculative decoding in SGLang with CUDA graphs on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. Savings depend on drafted-token acceptance, which is lower on conversation than code and lower on Romansh than English. Savings shrink as batch size grows, and block drafters can push verification into the compute-bound regime, where speculation raises energy per token.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026

SGLang announces Day-0 support for Mistral Large 4 ("Le Chonk"), saying it is the first engine to serve the model on NVIDIA Vera Rubin. The post describes the model as Mistral's largest to date: a natively multimodal 1T-parameter mixture-of-experts model covering instruct, reasoning, coding and agentic workflows. The fetched text is truncated, so benchmark claims and launch commands are not visible.

x.comthreadAdded 30 Oct 2026, published 29 Oct 2026

AnySpeechRecognizer and AnyDecisionModel are two Swift packages from Hugging Face. The first mirrors Apple's SpeechAnalyzer API with swappable engines (NVIDIA Parakeet, Alibaba Qwen3-ASR on Core ML) and adds Sortformer speaker diarization. The second returns typed yes/no, choice and score answers with probabilities via MLX, Core AI, llama.cpp, Jev or Ollama. The post gives M4 Max timings, model sizes and calibration options.

huggingface.coarticlePublished and added 29 Oct 2026

Mistral Large 4 (Le Chonk) is a multimodal, multilingual Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context, released under the Mistral Ecosystem Open-Weight License. The post covers its MLA and DeepSeek Sparse Attention architecture, a 1.6B Pixtral encoder, and benchmarks including 82% on the Artificial Analysis Cyber Index and 93% on Cybench. It gives run recipes for transformers, vLLM, llama.cpp, TRL fine-tuning and Inference Endpoints.

huggingface.coarticlePublished and added 29 Oct 2026

Decrypt reports, citing The Information, that OpenAI plans to buy Google TPUs rather than rent them, to reduce dependence on Nvidia. Neither company has confirmed, and price, volume and timing are undisclosed. The article covers OpenAI's TPU optimization job listing, its Broadcom-built Jalapeño inference chip due end of 2026, Nvidia and AMD deals, and Anthropic's multi-gigawatt TPU agreement.

decrypt.coarticlePublished and added 28 Oct 2026

QbD-VLA is a mixed-precision post-training quantization method for vision-language-action models. It uses task success from short closed-loop evaluations to assign weight and activation bit widths, accounting for interactions between components. On four VLA models across all four LIBERO suites it holds task performance at about 4-bit weights and under 6-bit activations. The gain on SIMPLER is small. On a Franka Research 3 arm, X-VLA shows no detected loss, but π0.5 fails.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

FlatCake is a KV cache storage architecture for LLM serving that treats host DRAM and NVMe SSD as peer media in one 64-bit address space, instead of a DRAM-over-SSD tier. Blocks are placed once, in session-contiguous, layer-major order, and streamed via GPUDirect Storage. On a Claude Code trace with DeepSeek-V4-Flash it holds a 94% hit rate (Mooncake: 64%) and cuts peak-load mean TTFT 1.6x.

arxiv.orgpaperAdded 30 Oct 2026, published 28 Oct 2026

Survival LLM is a free, open-source desktop app for macOS (Apple Silicon) and 64-bit Windows that runs an AI model and an offline emergency library on the user's computer. It checks the machine's memory, downloads a fitting model, and answers from sources such as Wikipedia, WikiMed, Ready.gov and CDC guidance, citing each article. Users can add their own files; version 0.3.2 is dated October 2026.

brangus.iotoolAdded 28 Oct 2026, published 27 Oct 2026

Photon Capital's thread covers the 27 October 2026 NVIDIA and Oracle announcement of general availability of Vera Rubin NVL72 and Vera CPUs on OCI. It lists specs (Spectrum-6 at 102.4Tbps, HGX Rubin NVL8 with Xeon 6, 150,000 initial Vera CPUs) and OpenAI's use of Vera CPUs for agentic sandboxes. The commentary argues CPU capacity can cut cost per task and shorten KV cache residency in GPU memory.

x.comthreadAdded 28 Oct 2026, published 27 Oct 2026

Metacognitive Decoding is a lightweight plug-in controller for diffusion language model backbones. It learns when to finalize tokens and how far generation should extend, and is trained on agent trajectories to balance correctness against generation cost. Across five benchmarks it cuts inference FLOPs by 20.6% on average versus the most efficient baseline, with competitive task accuracy. The abstract names no specific benchmarks or backbones.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

LFM3 is Liquid AI's new family of open-weight multimodal models, six models of about 2B parameters branched from LFM3-1.7B-Base (22T pre-training tokens, 256K context). The hybrid backbone uses 13 gated short-convolution, 5 sliding-window attention and 6 global attention layers. Variants cover text, vision with video, audio, omni and two decision models, with on-device speed numbers, fine-tuning demos and robotics extensions.

liquid.aiarticleAdded 28 Oct 2026, published 27 Oct 2026

CRISP is a combination-aware framework that co-designs pruning and runtime scheduling for Mixture-of-Experts inference on memory-constrained edge platforms. It reorders neurons by importance so one stored expert supports several execution widths, gives hot, warm and cold expert combinations different widths, and overlaps expert loading with computation. On Mixtral-8x7B and DeepSeek-MoE-16B it raises throughput up to 2.1x at the same memory budget, with accuracy loss limited to 3.6%.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

Walkthrough of running OpenCode against Qwen3.5 9B served by Ollama on a Mac, with Wi-Fi switched off. The author fixes a one-line accumulation bug in an expense calculator and confirms all five unittest tests pass. It covers the opencode.json setup, memory budgeting, permission prompts and a pytest-versus-unittest detour. Two further exercises (CSV report, log analyzer) were not yet run offline.

latent.spacewriteupPublished and added 27 Oct 2026

p-DCIM is a 28 nm SRAM digital compute-in-memory macro (0.16 mm2, 0.41 mW) that works as an Ising machine for RAG evidence selection. Selection is cast as a QUBO problem, and the same MAC datapath computes embedding similarities and annealing steps. Replacing LLM reranking with it gives a selection-energy estimate over six orders of magnitude lower and cuts total energy per query by 67%, with comparable token-level F1 on the evaluated examples.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

DCKV is a differentiable codec for the prompt KV cache of hybrid language models, which keep a KV cache only in their few full-attention layers. Transform, quantizer and entropy models (conditioned on prompt tokens and already decoded neighbouring positions) are trained end to end against a frozen model. On Qwen3.5-2B it stores a token in over 55 times fewer bytes than bf16 with at most 1.8 points lost on three long-context benchmarks, and over 100 times fewer with at most 3.6 points lost.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

AxisQ is a post-training quantization framework for camouflaged object detection (COD) that allocates activation bits by feature depth. It uses the channel domain for shallow features and the local-frequency domain for deep features, under a fixed per-layer average bit budget. It adds a boundary-aware Hessian approximation for reconstruction. On CAMO at W2A2, weighted F-measure improves 12.7% for CFRN and 33.3% for CamoFormer-P over the strongest baselines.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

AINews issue from Latent Space on 27 October 2026, led by Anthropic's S-1 filing for a Nasdaq listing (ticker ANT). It reports H1 2026 revenue of $16.3B, an 85% gross margin, $974M Q2 operating profit and 48% of revenue through cloud marketplaces, with reactions from named analysts. It also covers model releases, inference economics and agent tooling. The page text is truncated.

latent.spacearticlePublished and added 27 Oct 2026

OblivDGL is a framework for secure GNN training and inference in machine-learning-as-a-service settings, modelled on Deep Graph Library. It uses an oblivious message passing paradigm and a new oblivious graph representation built on function secret sharing, revealing only the total node and edge counts. On four benchmark graphs it cuts training communication up to 1024x with up to 14.3x speedup, and inference communication up to 754x with 4.19x speedup.

eprint.iacr.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

The paper folds the per-key logit bias of Attention Matching (AM) KV cache compaction into the keys along one closed-form direction per head, fitted on context queries. The result is a plain cache that FlashAttention and vLLM paged caches can use, with output error bounded by how far each query's projection on that direction is from 1. Unmodified vLLM serves folded caches through a KV connector. On QuALITY and LongHealth the fold is within noise of AM in 13 of 14 settings. Under AM's published per-head budgets it is within noise in 9 of 10.

arxiv.orgpaperAdded 28 Oct 2026, published 27 Oct 2026

Cloudflare's postmortem of a tenant isolation bug in the GLM-5.3 model on Workers AI, active 27 August to 9 September 2026. Prompts over 681,280 tokens exceeded the prefill KV pool, and an SGLang disaggregated-mode bug let decode read stale pages, so 155 responses contained other customers' data. It covers the timeline, config and engine fixes (SGLang v0.5.22), and recommendations for operators.

blog.cloudflare.comwriteupAdded 27 Oct 2026, published 26 Oct 2026

Ollama overview thread by Akash Rajagopal on X. It lists six steps: install on macOS, Linux, Windows or Docker; run `ollama run gemma4`; use the REST API on localhost:11434 (/api/chat); call it from ollama-python and ollama-js; llama.cpp as the README's supported backend; and `ollama launch claude` to connect Claude Code. Codex, OpenCode, Copilot CLI and Droid also work. The thread cites 184k GitHub stars and an MIT licence.

x.comthreadAdded 27 Oct 2026, published 26 Oct 2026

Hob is an MIT-licensed desktop app from Frigade that runs open-weight models such as Gemma and gpt-oss locally on Mac, Windows and Linux. It picks the best open-weight model that fits the user's computer and sets it up. It is positioned as a free, private, offline alternative to Claude and ChatGPT for tasks like filling out PDFs and working through documents. The page gives no technical detail on its runtime.

github.comrepoAdded 27 Oct 2026, published 26 Oct 2026

Ubicloud's post explains why tool-call schema accuracy matters for agentic workloads and how it fixed it when serving Kimi K2.6 and DeepSeek V4 Flash. With vLLM 0.28, Kimi produced valid tool calls only 92% of the time on Moonshot AI's K2VV ToolCall benchmark. Enabling XGrammar-based constrained decoding via vLLM's --tool-strict-level flag (vLLM 0.31) raised schema accuracy to 100%.

ubicloud.comwriteupAdded 27 Oct 2026, published 26 Oct 2026

SemiAnalysis claims AMD chips running vLLM can earn up to $3 per chip-hour in profit serving the open-source Kimi K3 model, even after higher chip rental prices following Claude Code's rise. Kimi K3 has over 2.8 trillion parameters and fits AMD's 288GB per chip, while B200 cannot hold it in one server. B200 must then use tensor or pipeline parallelism over 400Gbit/s InfiniBand or Ethernet, which SemiAnalysis says is 18x slower than NVLink. The post gives no methodology.

x.comthreadAdded 26 Oct 2026, published 25 Oct 2026

ZipFusion is a lossless KV cache compression scheme whose attention kernel reads the compressed cache directly and rebuilds the original values in registers with simple bit operations, so no separate decompression pass is needed. It also reassigns the memory freed by compression across layers to hold more KV cache. Built on FlashInfer and integrated with SGLang, it gives about 1.25x decode attention speedup on an H100 and up to 26.3% higher decode throughput at 8K context than uncompressed SGLang. It reaches up to 2.76x the throughput of the best lossless KV compression baseline.

arxiv.orgpaperAdded 27 Oct 2026, published 24 Oct 2026

RAQ-MLA (Role-Aware Quantization for Multi-Head Latent Attention Caches) is a training-free method for quantizing the MLA KV cache. It fits a separate invertible transform to the compressed latent (tuned for attention-weighted value error) and to the positional key (tuned for score perturbation). It reaches about 2.5 bits per element, a 6.5x gain in theoretical KV-cache capacity, with near-lossless reasoning and coding results on four MLA models.

arxiv.orgpaperAdded 27 Oct 2026, published 24 Oct 2026

Building Reasoning Models from the Ground Up is a series of eight coding videos (about 11 hours) by Sebastian Raschka, accompanying his book Build a Reasoning Model (From Scratch). Starting from a pre-trained Qwen3 0.6B model, it implements KV-cached generation, a MATH-500 verifier, self-consistency and self-refinement, simplified GRPO for RLVR, and distillation from DeepSeek R1 answers in Python and PyTorch.

magazine.sebastianraschka.comcoursePublished and added 24 Oct 2026

DualPace is an LLM serving runtime that treats the host as an independently scheduled decode engine. A GPU engine runs prefills and GPU-resident decodes, while an AMX-enabled CPU engine batches and advances CPU-resident decodes, optionally calling GPU Dense Assist for dense stages. In decode-oriented scenarios it raises throughput 6.9 to 43.2% over NEO and 11.5 to 90.6% over vLLM at similar latency.

arxiv.orgpaperAdded 27 Oct 2026, published 24 Oct 2026

SemiAnalysis reviews AWS Trainium3 and the new TorchNeuron native PyTorch backend after a few weeks of hands-on use. It reports TorchTitan Qwen3-32B results on Trainium2 (11,099 tokens/s eager at 23.89% MFU, 14,156 compiled at 30.47%), a 27-minute gpt-oss compile time, 89 of 96 FlexAttention cases passing, matmul comparisons with GB300 and B200, and a positive view of the Neuron Explorer profiler.

newsletter.semianalysis.comarticleAdded 29 Oct 2026, published 23 Oct 2026

Accordion is a mixture-of-experts language model (Accordion-6B-A0.5B) that supports five inference budgets through fixed-prefix layer looping, with the minimum budget being the unlooped model. After 3.6T pretraining tokens, deeper inference raises the macro-average over 37 tasks by 3.01 points, and the middle budget captures 89% of that gain with half the extra layer executions. SFT and RL keep the budgets, merging three domain experts via parameter-group merging and budget-matched multi-teacher on-policy distillation.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

BiLoCo is a post-training quantization method for FP4 LLMs that adds low-rank corrections whose left and right factors are stored as binary ±1 sign vectors instead of BF16. This uses 16x less storage per rank-one component and lets the correction run on CUDA cores alongside the FP4 tensor-core GEMM. On 4B to 72B LLMs it matches or improves memory-matched SVDQuant. A fused B300 kernel is at most 11% slower than FP4-only GEMM at batch size 1 for ranks up to 512.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

MPQ (Message-Passing Quantization) is a post-training quantization method that treats row-wise PTQ as posterior inference over discrete codebooks. It keeps soft posteriors across weight blocks using vector approximate message passing, then refines a hard-decision initialization with exact block coordinate descent. Tested at 2-bit lattice and 3- and 4-bit scalar quantization on Llama, Qwen and Mistral models, it improves perplexity and zero-shot accuracy over GPTQ-style baselines.

arxiv.orgpaperAdded 26 Oct 2026, published 23 Oct 2026

AKV introduces cache engineering, which lets LLM agents control their active KV cache with drop (exclude cache from future attention) and repos (adjust retained keys' position encodings) without editing the trajectory or re-prefilling. The AKV serving engine keeps a stateless API while managing stateful cache reuse, eviction and restoration. A tool-response rolling-drop policy raised benchmark scores by 1.7 points and cut cost 27.7% versus threshold-triggered summarization.

huggingface.copaperAdded 23 Oct 2026

The torch-neuronx model support Space is a Hugging Face and AWS Neuron lookup that shows whether a Transformers model runs unmodified on AWS Trainium2. Each result lists core count, wall time and peak memory, plus the run.py script and a public NEFF compile cache. Tests generate 32 tokens on up to 16 cores. Results are in the aws-neuron/torch-neuronx-model-support Parquet dataset.

huggingface.coarticlePublished and added 23 Oct 2026

BeNice! is a post-training method for looped language models that broadcasts the final recurrence's KV-cache into the first recurrence, so one cache can serve all recurrences. It follows a first stage that uses stochastic depth in pretraining to induce contractive residual dynamics. It matches standard finetuning's unshared validation perplexity and composes with multi-latent attention, Kimi delta attention and fp8, mxfp8 and nvfp4 cache quantization. In simulated low-memory long-context inference it raises cache hit rates by up to 31.4% and throughput by up to 23.6%, and cuts p90 time-to-first-token by up to 83.6%.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

Latent Space interview with Modal CEO Erik Bernhardsson, after Modal's $750M Series D at a $16B valuation and passing $1B annualized run rate. He explains why GPU demand is not "rectangular" like fixed-term capacity contracts, how Modal pools demand and buys from about 25 clouds on three-year deals, and why sandboxes grew to about a third of revenue. Also covers GPU price-collapse risk and non-NVIDIA accelerators. Only the first part of the transcript was available.

latent.spacetalkPublished and added 22 Oct 2026

Latent Space interview with Modal CEO Erik Bernhardsson after the company's $750M Series D at a $16B valuation, with a transcript. He explains why GPU demand does not match rectangular fixed-term capacity contracts and how Modal pools demand across customers. He covers sourcing from nearly 25 clouds on multi-year deals, the risk of falling GPU prices, sandboxes reaching about a third of revenue, and non-NVIDIA accelerators.

latent.spacetalkAdded 23 Oct 2026, published 22 Oct 2026

This October 2026 arXiv paper evaluates four LLM inference optimization families on Apple Silicon: KV-cache pruning, speculative decoding, ported GPU kernels and post-training quantization. Tests run on four Apple machines with a single-request decode workload, plus matched RTX 4090 runs. KV-cache pruning reaches 1.75x throughput on Apple, all tested speculative decoding methods reduce throughput, and ported kernels give up to 1.20x.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

Rate-Controlled Refresh (RCR) is a KV cache re-selection method for multimodal LLMs in long-form generation. Attention drift decides when and where to re-select visual tokens, and a per-sample controller sets how often. RCR beats InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings and stays within 1.7 points of exact per-step re-selection. On video description, where drift is homogeneous, it no longer beats a fixed threshold.

arxiv.orgpaperAdded 24 Oct 2026, published 22 Oct 2026

OnKey is a free, MIT-licensed singing trainer for Apple Silicon Macs. After a song ends in Spotify, Apple Music or browser YouTube, it separates the lead vocal with UVR's MDX-Net Karaoke 2, tracks pitch with CREPE and turns it into syllable-level notes with the GAME model. On later plays it scrolls the notes, draws the user's live pitch and scores each line. Analysis runs on-device, and the repo explains the build steps and network requests.

github.comrepoAdded 24 Oct 2026, published 22 Oct 2026

SARIA (Selective Admission and Reload for Inference Acceleration) is a KV caching system that jointly manages DRAM and SSD for multi-turn LLM agents. It persists only blocks likely to be reused beyond DRAM residency, promotes SSD-hit blocks into DRAM for same-session reuse, and paces DRAM reads with a transfer coordinator. Built on LMCache MP, it cuts SSD writes by 57.1 to 58.9% versus full duplicate caching and mean TTFT by 42.3 to 79.7% versus DRAM-only caching.

arxiv.orgpaperAdded 23 Oct 2026, published 22 Oct 2026

RIFT is a family of selection-based KV cache compression strategies for speculative-decoding draft models. It targets "decode staleness", where prefill-focused selection loses relevance as output grows, by balancing prefill information with recent generated tokens, refreshing the selection adaptively, and training drafters with sparse attention masks. Qwen3.5-4B drafting for Qwen3.5-122B keeps over 98% of full-cache acceptance length at 32K input, 4K output and an 8K KV budget.

arxiv.orgpaperAdded 23 Oct 2026, published 22 Oct 2026

AINews issue for 20-21 October 2026. The top story covers PHP2Shell (CVE-2026-109618 and CVE-2026-109619), two PHP-engine memory-corruption bugs found separately by Calif researchers using GPT-5.6 Sol under OpenAI's Patch the Planet and by Wiz, followed by two months of coordinated remediation. It also covers the Flint coding MoE, DeepSeek's DeepInfer, TorchTPU, eval critiques and agent product launches. The text is truncated.

latent.spacearticlePublished and added 22 Oct 2026

Gandalf is an SSD KV cache offloading system for LLM serving that is aware of SSD lifetime. It prioritizes admitting full-attention KV cache to SSD over cache from other attention types, using application-level information rather than access history alone. The authors report a 7.3x average SSD lifetime extension with negligible cache hit rate loss across several workload traces and hardware capacities. The design may extend to High Bandwidth Flash.

arxiv.orgpaperAdded 23 Oct 2026, published 22 Oct 2026

Flint is the first model from Recursive SI, a fast local coding model with open weights released the same day. It was built with the company's automated research system, the Eureka Machine. The post claims up to 2x faster speed than comparable models: 125+ tokens/sec on a MacBook Pro and 150+ on RTX Spark. The captured text is truncated and gives no benchmarks or model size.

x.comthreadAdded 22 Oct 2026, published 21 Oct 2026

VocabX is a vocabulary adaptation method from Dharma-AI that selects sub-word and multi-word "macro" tokens by serving cost instead of frequency. On a Qwen3-4B Portuguese essay grader, response length fell from 350 to 101 tokens, the LM head shrank from about 152,000 to 65,000 rows, and measured cost dropped 77.0% on an RTX 5090 with vLLM. The bench score rose from 0.703 to 0.810. It beat FVT, MWT and in-place BPE across 15 model and domain combinations.

huggingface.coarticlePublished and added 21 Oct 2026

The Huawei-DeepSeek Symbiosis analyzes DeepSeek's 30 September 2026 Ascend support for TileLang, DeepGEMM, DeepEP, TileKernels, FlashMLA and DeepSelect. It microbenchmarks the Ascend 950PR (BF16 GEMM, HBM bandwidth, die topology, AICPU) against H100, and profiles DeepSeek-V4.1-Flash on 32 Ascend 950DT chips. It also lists vLLM-Ascend gaps on x86 hosts.

newsletter.semianalysis.comarticleAdded 29 Oct 2026, published 21 Oct 2026

SemiAnalysis thread arguing that vLLM with Wide-EP and EPLB can reproduce DeepSeek's serving performance. It reports decode throughput within 6% of DeepSeek's DeepInfer release and about $39 per GPU-hour revenue at DeepSeek's off-peak pricing, at 200 P90 TPS on the AgentX workload. It explains why DeepInfer is not on InferenceX: the release supports only offline, single-turn inference. It also estimates GB300 profit at OpenRouter prices.

x.comthreadAdded 22 Oct 2026, published 21 Oct 2026

Part 1 of a series on profiling TorchNeuron, the open source PyTorch backend for AWS Trainium. It shows how to configure torch.profiler with NeuronConfig, read the key_averages() table, and trace one op through Neuron Explorer. Three RMSNorm variants are compared: naive eager (2.355 ms device time), torch.rms_norm (403 μs) and torch.compile (375 μs).

huggingface.cowriteupPublished and added 21 Oct 2026

Protos reports on the AMD and Anthropic compute deal: Anthropic commits to buy over $20 billion of AMD AI compute, up to 2 GW of Instinct MI450 chips from the first half of 2027, while AMD commits up to $5 billion to Anthropic stock through fiscal 2028. It cites WSJ reporting that hyperscalers cap Anthropic's compute, and Anthropic's 2025 operating loss above $8 billion on $4.6 billion revenue.

protos.comarticlePublished and added 21 Oct 2026

AINews issue for 19 to 20 October 2026 whose top story is DeepSeek's open-source DeepInfer, the inference engine behind V4.1 Flash. It covers prefill/decode separation, wide expert parallelism, a 3FS-backed prompt cache with 96% input hits, vLLM's decode result within 6% of DeepInfer on GB200, and disputed margin estimates built on the reported $38 per accelerator-hour. It also digests other news, and the page text is truncated.

latent.spacearticleAdded 23 Oct 2026, published 21 Oct 2026

This AINews issue (20 October 2026) leads with DeepSeek's open-sourced DeepInfer, the vLLM-based inference engine behind V4.1 Flash. It covers prefill/decode separation, a 3FS-backed prefix cache with 96% hit rate, SWA Bounded Replay, and DeepSeek's claims of 155T tokens/day and $38 revenue per accelerator-hour. It also covers vLLM's within-6% decode reproduction on GB200, disputed margin estimates, and a round-up of other AI news. Only the first 40,000 of 56,439 characters were available.

latent.spacearticlePublished and added 21 Oct 2026

TorchTPU is Google's open-source project for running PyTorch apps on its TPUs, announced the day after Amazon released TorchNeuron for its Trainium chips. Google says migration needs "a simple configuration change" with no rewrite. It worked with Meta, Hugging Face, and the vLLM and SGLang teams on compatibility, and added an agent suite that helps migrate apps and fix errors. The post is brief and gives no benchmarks.

x.comthreadAdded 21 Oct 2026, published 20 Oct 2026

Latent Space interview with Evan Conrad, CEO of SF Compute, recorded at AI Engineer NYC. He explains why the company runs a cloud's operations to physically settle GPU contracts on an order book with multi-constraint matching. Topics include preemption and storage recovery, hardware cohorts, batch versus live inference pricing, forward curves, a B300 price arbitrage, and cash versus physical settlement. The last ~3,600 characters of the transcript were not available.

latent.spacetalkAdded 23 Oct 2026, published 20 Oct 2026

Cache Engineering for Agents proposes letting LLM agents control their active KV cache directly, using a drop operation to exclude cache from attention and repos to adjust retained keys' position encodings, without editing the trajectory or re-prefilling. The authors build the AKV serving engine to handle history-dependent cache states. A tool-response rolling-drop policy raises average benchmark scores by 1.7 points and cuts cost by 27.7% against threshold-triggered summarization. AKV gives up to 1.2x output throughput and 1.5x faster P95 latency.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

This paper studies 2-bit post-training quantization (PTQ) of six dense and mixture-of-experts models from 3.8B to 32B parameters. It finds the quantization representation matters most: vector codebooks give usable models where scalar PTQ falls to chance. Calibration and recovery data must be chosen jointly. The recipe uses about 2.3 effective bits per weight, keeps 89 to 97% of full-precision QA accuracy, and needs 1.4 to 30x less compute than the strongest QAT baseline.

arxiv.orgpaperAdded 22 Oct 2026, published 20 Oct 2026

Interconnects' Latest open artifacts (#25) covers open-weight AI in fall 2026. Policy items: China's MOFCOM proposal for pre-approval of international model sharing, and a DHS order barring DeepSeek from federal civilian procurement. DeepSeek released its internal inference stack DeepInfer. Picks include MiMo-V2.6-Pro-RL, DeepSeek-V4.1-Flash-1017 (552B-A16B, MIT), Clef, Nemotron-3.5-Super-VL and Reflection's Beam-1.0 (Apache 2.0).

interconnects.aiarticlePublished and added 20 Oct 2026

DeepInfer is DeepSeek's open-source inference framework for offline inference of DeepSeek-V4.1-Flash, released under Apache 2.0. It covers attention data parallelism, MoE expert parallelism with load balancing, draft-token decoding with confidence-scheduled verification, batch-invariant inference, deterministic sampling and structured outputs. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The KV cache system, prefill/decode disaggregation and load balancing are not included.

github.comrepoAdded 21 Oct 2026, published 20 Oct 2026

AINews issue for 16 to 19 October 2026 from Latent Space, a digest of Twitter and Reddit AI news. It covers DeepSeek V4.1 Flash 1017 (552B total, 16B active MoE, AA Intelligence Index 45), MiniMax M3.1 Flash, Mercury 3, Meta Muse, Google's Harness², vLLM v0.32.0 and a Terminal-Bench audit. Only the first 40,000 of 47,045 characters were available.

latent.spacearticleAdded 30 Oct 2026, published 20 Oct 2026

AINews issue covering 16-19 October 2026 from Latent Space, summarising 12 subreddits and 544 Twitter accounts. It leads with DeepSeek V4.1 Flash 1017 (552B total, 16B active MoE, 1M context, MIT license, AA Intelligence Index 45), plus MiniMax M3.1 Flash, Mercury 3 and Meta Muse. It also covers vLLM v0.32.0, InferenceX results, Harness² and Terminal-Bench audit findings.

latent.spacearticlePublished and added 20 Oct 2026

AnyLanguageModel 1.0 is a Swift package that gives apps Apple's Foundation Models API across the system model, llama.cpp, MLX, Core ML and hosted providers. This release post covers tool calling and guided generation on every provider, image input, and hosted support for OpenAI, Anthropic, Gemini, Ollama and Hugging Face Inference Providers. It also ships swift-huggingface 1.0 and swift-xet 1.0, and lists small open-weight models that fit on iPhone and Mac.

huggingface.coarticlePublished and added 20 Oct 2026

PyTorch 2.15 MPS release post from the Hugging Face PyTorch team, covering the Apple silicon backend. It adds float8_e4m3fn tensors with F.scaled_mm, a native grouped matmul that speeds MoE decoding 5 to 6.5x (Qwen3.6-35B-A3B from 9.4 to 61.3 tokens/s on an M3 Ultra), and an 8x faster softmax. Wan2.2 VAE decode peak memory falls from 270 GiB to 8 GiB. It previews 2.16 attention on Metal.

huggingface.coarticlePublished and added 20 Oct 2026

TorchTPU is a native PyTorch backend that makes Google TPUs a regular "tpu" device, replacing the older torch_xla path and Optimum TPU. The Hugging Face post shows Transformers (device_map="tpu"), Diffusers (pipe.to("tpu")) and Pallas kernels from the Kernels Hub running on it. It reports up to 21.5x faster decoding on Qwen3-30B-A3B at batch 64 and a 96.5% TPU test pass rate.

huggingface.coarticlePublished and added 20 Oct 2026

Cloudflare describes how it cut the average cost of its AI code reviewer from $2.43 to $0.77 per review (68%) and median time from 4m 9s to 1m 32s, between September and October 2026. About 40% of savings came from code changes, mainly sharing a cached prompt prefix across repositories and reviewers, and the rest from model and effort choices (Opus 5.5, Sonnet 5.5). Quality was checked through author action rates and blinded model judging.

blog.cloudflare.comwriteupAdded 29 Oct 2026, published 19 Oct 2026

TokenGate is a small drop-in component that adds end-to-end learnable token pruning to trained LLMs for long-context prefill. It uses token-gated attention, inspired by Mixture-of-Experts, to re-weight attention logits with token importance scores, which gets gradients through the non-differentiable top-K step. The authors report 1.4-11.3x faster time-to-first token and 1.5-3.4x lower peak memory with no accuracy loss, on Qwen3 and Qwen3.5.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

Tandem is a runtime for serving hybrid linear-attention GUI agents on mobile NPUs. It saves recurrent state where consecutive prompts diverge, keeps most prefill on the matrix unit, uses a verification kernel that reads each weight block once, and releases NPU power votes between bursts. On a Snapdragon 8 Elite it cuts per-step latency 30% for text screen descriptions and 16% for screenshots, cuts energy per text step by 33%, and lowers loaded-model standby power from 0.76 to 0.22 W.

arxiv.orgpaperAdded 21 Oct 2026, published 19 Oct 2026

Maxwell (Maxwell's Daemon) is doubleAI's first "code execution model" (CEM). It runs inside a live program, finds bottlenecks and emits faster machine code for the exact CPU or GPU, leaving the source untouched. Each rewrite ships with a machine-checked proof of semantic equivalence, built on a formalisation of Python. Early access is announced; the post gives no benchmark figures.

heymaxwell.aiarticleAdded 20 Oct 2026, published 19 Oct 2026

TorchNeuron is an AWS PyTorch plugin that exposes Trainium as a native "neuron" device with eager mode, torch.compile, DTensor and FSDP2, entering public beta on 20 October 2026 (pip install torch-neuronx). The post shows Transformers text generation with tensor parallelism, TRL SFT of Qwen3-8B on 32 NeuronCores (TP 4 x FSDP 8), and Diffusers TP inference. It says 89% of the top 100 text-generation architectures work, versus 46% with Optimum Neuron v0.4.5.

huggingface.coarticlePublished and added 19 Oct 2026

HERMES is a privacy-preserving inference (PI) system optimised for multi-token LLM decoding rather than prefill. It swaps high-round nonlinear protocols for compute- and bandwidth-heavy ones with fewer rounds, and adds a homomorphic-encryption matrix-multiplication protocol that keeps the KV cache encrypted on the server and uses grouped-query attention. On Llama 3.1 8B Instruct it cuts TPOT by up to 2.7× and total PI latency by up to 2.1×.

eprint.iacr.orgpaperAdded 19 Oct 2026, published 18 Oct 2026

Model card for DeepSeek-V4.1-Flash-1017, a minor update to DeepSeek-V4.1-Flash with MIT-licensed weights. It claims gains in reasoning, coding, agentic tasks and visual benchmarks, with shorter task time and output length that move the performance–cost Pareto frontier across agent harnesses (Claude Code for Terminal-Bench 4.0, Mini-SWE-V2 for DeepSWE v1.1). It ships no Jinja chat template. It provides a reference encoding.py for multi-turn chat, tool calling, thinking mode, numeric reasoning effort and interleaved images, plus deepseek-recipe, Rust libraries with Python bindings for prompt encoding and output parsing. Serving recipes are given for Transformers, vLLM, SGLang and Docker Model Runner.

huggingface.corepoAdded 18 Oct 2026, published 17 Oct 2026

AINews issue for 10/15–10/16/2026, covering Twitter and Reddit. It covers the open-weights releases Meta's Muse Spark Open (816B/42B-active MoE, AA Intelligence Index 39, custom license) and Reflection Beam 1.0 (501B/23B-active, Apache 2.0), plus NVIDIA Nemotron 3.5, StepFun Step 5 Preview and FLUX 3 Image Klein. It also covers Gemini 4 Argon, Haiku 5.5 pricing, "decision model" classifiers (Liquid d1-ultra, Perplexity pplx-decider), TPUv7/Vera Rubin/MI355X serving benchmarks, interpretability claims (Qwen3-32B neuron audit), agent research, eval-integrity and safety disclosures, and China export-control proposals. The page also announces a pilot video show, High Taste Testers Ep. 1.

latent.spacearticlePublished and added 17 Oct 2026

AINews issue for 15-16 October 2026, from Latent Space, summarising 12 subreddits and 544 Twitter accounts. It covers Meta's Muse Spark Open (816B MoE) and Reflection Beam 1.0 with Artificial Analysis scores and pricing. It also covers Gemini 4 Argon, decision models, TPUv7 and Vera Rubin serving benchmarks, interpretability claims, agent papers, and China's proposed open-source export rules.

latent.spacearticleAdded 30 Oct 2026, published 17 Oct 2026

Llmspeed is an open-source web tool that lets you get a feel for how fast a local LLM runs on a given device before buying hardware. It shows estimates for a short prompt on a single device, using measured numbers from community benchmarks (r/LocalLLaMA, localscore.ai) and filling gaps with figures derived from similar setups or projected from specs. It helps with local-LLM hardware purchasing decisions.

llmspeed.devtoolAdded 18 Oct 2026, published 17 Oct 2026

Model card for Meta Superintelligence Labs' Muse Spark Open, an open-weights, natively multimodal model with 816B total and 42B active parameters and a 1M-token context window. It targets coding and agentic work (tool use, long-horizon tasks, multimodal reasoning, synthetic data generation, LLM-as-judge). It ships in BF16 and NVFP4, needs about 700GB VRAM (NVFP4) or 2TB (BF16), runs on SGLang, vLLM, Transformers, TensorRT-LLM, Unsloth, llama.cpp and MLX, and supports a reasoning_effort setting. The Muse Model License requires "Built with Muse Spark Open" attribution and a separate agreement for Model-as-a-Service above $20M revenue. The card reports safety results against Kimi K3 on cyber, chem/bio and loss-of-control evals.

huggingface.corepoAdded 17 Oct 2026

PS5LM is a homebrew app for jailbroken PS5s (firmware 7.00–13.60). It runs llama.cpp with the ggml-vulkan backend on the console's RDNA 2 GPU, using 11.4 GB of shared memory, and loads any GGUF model. It serves OpenAI /v1/chat/completions, Anthropic /v1/messages (usable from Claude Code, Codex and opencode) and a browser chat page on port 8081. It also supports image-projector vision models and TypeSafe-style decision models through a /v1/systemone endpoint. A planner picks the longest context and best cache that fit the free memory, and the app downloads models from Hugging Face.

github.comrepoAdded 17 Oct 2026, published 16 Oct 2026

This paper shows that training-free relaxed (lossy) verification in speculative decoding causes length inflation: completions get longer, mostly through extended reasoning traces, which offsets gains in accepted tokens per round. Across twelve target-drafter pairs, relaxed rules accept 1.34x as many tokens per round but produce 1.54x longer completions. Rules that cut verifier rounds usually lose substantial accuracy, so the authors argue completion length and verifier rounds must be reported.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

FuTransformer is a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It uses FP8 prefill with fused GEMM epilogues and INT4 weights plus an INT4 KV-cache copy for decode, with a block-diagonal Hadamard transform to cut cache quantization error. On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8, and INT4 decode is 1.3-2.4x faster than the faster PyTorch baseline. Llama-3.1-8B decode takes 2.8 ms per step at batch one, versus 8.1 ms for compiled transformers. WikiText-2 perplexity rises from 6.24 to 7.23 with round-to-nearest INT4 weights, or 6.99 with GPTQ.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

DLite is a history-free speculative decoding framework for agentic LLM workloads. It reuses the target model's contextual hidden states and predicts locally without keeping the draft model's historical KV cache, which avoids the up to 80% acceptance-length drop seen with long, multi-turn contexts. A lightweight sequential head restores intra-block causal dependencies, and a two-stage training strategy is used. It reaches 2-6 acceptance length and beats DSpark by up to 70%.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

TileDRAM is a compiler-runtime toolchain that adds tile-to-DRAM layouts as a compiler-visible abstraction for compute-unit-local 3D DRAM GPUs. It extends GPU Linear Layout to DRAM placement and picks layouts per tile access signature using row-buffer behavior. On 108 MoE FFN cases in Accel-Sim/Ramulator2, row-conflict rate fell from 75.7% to 15.2%, with a 1.77x geomean speedup. Overhead on an RTX 3090 is 0.17%.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

A X thread benchmarking Qwen3.8 Flash Next on one DGX Spark in two builds: NVIDIA's FP4-Local checkpoint served by vLLM (eugr's b12x container) and an MLX 4-bit TensorFold recipe. The NVIDIA build scores higher on the author's agentic benchmark (81.2 vs 75.0), BFCL tool calling (82.4 vs 71.8 single-turn, 54.5 vs 43.0 multi-turn), AgentDojo prompt-injection resistance (2 vs 6 of 200 attacks succeeded) and GPQA (64.6 vs 59.1). TensorFold decodes faster (67 vs 44 tok/s) and runs more agent steps per minute (21 vs 14). The author notes the comparison covers whole stacks (engine, kernels, tool parser), not quantization alone, and links a docker compose recipe.

x.comthreadAdded 17 Oct 2026, published 16 Oct 2026

SpikeQ is a spike-guided post-training quantization framework for spiking neural networks on neuromorphic chips. It picks each weight level by simulating downstream neurons to minimize spike errors, and uses a reset-aware grid for pre-fire membrane potentials. With 4-bit weights and potentials, models stay within 2.59 points of full precision, with 7.8 to 7.9 times smaller memory and 6.4 to 6.9 times less energy.

arxiv.orgpaperAdded 20 Oct 2026, published 16 Oct 2026

MOSS-Image is an open-source text-to-image model family from OpenMOSS with a 5B backbone, released October 2026 as MOSS-Image-Base and MOSS-Image-SFT. It supports JSON layout control with bounding boxes, text rendering inside images, and prompts up to 4K tokens. The repo gives a Diffusers pipeline, a Gradio app, a ComfyUI plugin, LoRA training guidance, a prompt-enhancer skill and day-0 SGLang and vLLM-Omni support.

github.comrepoAdded 20 Oct 2026, published 16 Oct 2026

Announces Reflection's Beam-1.0-501B-A23B, an Apache 2.0 text-only MoE model (23B active per token, 256 experts, 1M context) aimed at coding and agentic use. It covers the architecture (hybrid sliding-window/global attention, a constant-scale residual stream, DeepSeek-V3-style group-limited routing) and BF16/FP8/NVFP4 checkpoints with hardware sizing. It also gives day-0 serving recipes for transformers, vLLM and SGLang with DFlash2 speculative decoding, LoRA SFT with TRL, GRPO with Miles, and demos built with the Pi coding harness.

huggingface.coarticlePublished and added 16 Oct 2026

Sentdex reacts to Artificial Analysis ranking NVIDIA's Nemotron 3.5 Super VL (120B total, 12B active MoE) as the top US open-weights model, scoring 26 on the Intelligence Index. That edges out Thinking Machines' Inkling Small (26), NVIDIA's Nemotron 3 Ultra (23) and Meta's Muse Glimmer (17). Sentdex's informal tests at vanilla settings show over 550 tok/sec, and they suspect it would rank higher on agentic coding. They see it as a candidate for a fast local coding agent model.

x.comthreadAdded 16 Oct 2026, published 15 Oct 2026