36 results for “SGLang”

SecurityWeek's weekly roundup of 30 October 2026 covers CVE-2026-97631, a 16-year-old buffer overflow in TianoCore EDK II that affects PXE network boot on most servers. It also covers a Cloudflare Workers AI cross-tenant leak caused by an SGLang bug, Anthropic's dispute with Cathedral over a bug bounty intrusion, and fake AI bots probing for secrets. Other items include AI-assisted ransomware (Gold Ogre, Aur0ra), the NIK C2 framework, and North Korean IT worker sanctions.

securityweek.comarticlePublished and added 30 Oct 2026🛡️ Security

This paper measures the energy cost of speculative decoding in SGLang with CUDA graphs on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. Savings depend on drafted-token acceptance, which is lower on conversation than code and lower on Romansh than English. Savings shrink as batch size grows, and block drafters can push verification into the compute-bound regime, where speculation raises energy per token.

arxiv.orgpaperAdded 31 Oct 2026, published 29 Oct 2026🧠 AI

SGLang announces Day-0 support for Mistral Large 4 ("Le Chonk"), saying it is the first engine to serve the model on NVIDIA Vera Rubin. The post describes the model as Mistral's largest to date: a natively multimodal 1T-parameter mixture-of-experts model covering instruct, reasoning, coding and agentic workflows. The fetched text is truncated, so benchmark claims and launch commands are not visible.

x.comthreadAdded 30 Oct 2026, published 29 Oct 2026🧠 AI

Cloudflare's postmortem of a tenant isolation bug in the GLM-5.3 model on Workers AI, active 27 August to 9 September 2026. Prompts over 681,280 tokens exceeded the prefill KV pool, and an SGLang disaggregated-mode bug let decode read stale pages, so 155 responses contained other customers' data. It covers the timeline, config and engine fixes (SGLang v0.5.22), and recommendations for operators.

blog.cloudflare.comwriteupAdded 27 Oct 2026, published 26 Oct 2026🧠 AI

Cloudflare postmortem of a tenant isolation failure in the GLM-5.3 Workers AI model (27 August to 9 September 2026), in which 155 responses to prompts over 681,280 tokens contained other customers' data. Root cause was a KV pool smaller than the 1,048,576-token context window, combined with an SGLang bug in disaggregated prefill/decode mode that let decode read stale KV pages. It lists the fixes (SGLang v0.5.22 patches), hardening steps and operator recommendations.

blog.cloudflare.comwriteupAdded 27 Oct 2026, published 26 Oct 2026🛡️ Security

Nanbeige4.3-33B is a 33.1B dense, Llama-style open model under Apache-2.0, released with a base model on Hugging Face. In Nanbeige's own table it beats Qwen3.8-27B on Terminal-Bench 2.1 (80.9 vs 73.0), DeepSWE (65.5 vs 42.5) and Automation Bench, but trails on SWE-Bench Pro, GDPval and GPQA Diamond. BF16 weights are about 66 GB, split across 4 GPUs in the vLLM and SGLang examples. No independent evals exist yet.

x.comthreadAdded 26 Oct 2026, published 25 Oct 2026🧠 AI

ZipFusion is a lossless KV cache compression scheme whose attention kernel reads the compressed cache directly and rebuilds the original values in registers with simple bit operations, so no separate decompression pass is needed. It also reassigns the memory freed by compression across layers to hold more KV cache. Built on FlashInfer and integrated with SGLang, it gives about 1.25x decode attention speedup on an H100 and up to 26.3% higher decode throughput at 8K context than uncompressed SGLang. It reaches up to 2.76x the throughput of the best lossless KV compression baseline.

arxiv.orgpaperAdded 27 Oct 2026, published 24 Oct 2026🧠 AI

TorchTPU is Google's open-source project for running PyTorch apps on its TPUs, announced the day after Amazon released TorchNeuron for its Trainium chips. Google says migration needs "a simple configuration change" with no rewrite. It worked with Meta, Hugging Face, and the vLLM and SGLang teams on compatibility, and added an agent suite that helps migrate apps and fix errors. The post is brief and gives no benchmarks.

x.comthreadAdded 21 Oct 2026, published 20 Oct 2026🧠 AI

Model card for DeepSeek-V4.1-Flash-1017, a minor update to DeepSeek-V4.1-Flash with MIT-licensed weights. It claims gains in reasoning, coding, agentic tasks and visual benchmarks, with shorter task time and output length that move the performance–cost Pareto frontier across agent harnesses (Claude Code for Terminal-Bench 4.0, Mini-SWE-V2 for DeepSWE v1.1). It ships no Jinja chat template. It provides a reference encoding.py for multi-turn chat, tool calling, thinking mode, numeric reasoning effort and interleaved images, plus deepseek-recipe, Rust libraries with Python bindings for prompt encoding and output parsing. Serving recipes are given for Transformers, vLLM, SGLang and Docker Model Runner.

huggingface.corepoAdded 18 Oct 2026, published 17 Oct 2026🧠 AI

A social post reporting that Reflection AI released the full weights of Beam, a 501B-parameter mixture-of-experts agentic coding model (23B active, 1M-token context), on Hugging Face under Apache 2.0. It claims RL training that penalizes wasted tokens gives roughly 3-4x less inference compute than GLM-5.2, and 80.9 on SWE-bench Verified. Launch support is listed for AWS SageMaker, vLLM and SGLang. The post notes the token-efficiency claims still need independent confirmation and that self-hosting needs about 500GB of GPU memory at 8-bit.

x.comthreadAdded 18 Oct 2026, published 17 Oct 2026🧠 AI

Model card for Meta Superintelligence Labs' Muse Spark Open, an open-weights, natively multimodal model with 816B total and 42B active parameters and a 1M-token context window. It targets coding and agentic work (tool use, long-horizon tasks, multimodal reasoning, synthetic data generation, LLM-as-judge). It ships in BF16 and NVFP4, needs about 700GB VRAM (NVFP4) or 2TB (BF16), runs on SGLang, vLLM, Transformers, TensorRT-LLM, Unsloth, llama.cpp and MLX, and supports a reasoning_effort setting. The Muse Model License requires "Built with Muse Spark Open" attribution and a separate agreement for Model-as-a-Service above $20M revenue. The card reports safety results against Kimi K3 on cyber, chem/bio and loss-of-control evals.

huggingface.corepoAdded 17 Oct 2026🧠 AI

MOSS-Image is an open-source text-to-image model family from OpenMOSS with a 5B backbone, released October 2026 as MOSS-Image-Base and MOSS-Image-SFT. It supports JSON layout control with bounding boxes, text rendering inside images, and prompts up to 4K tokens. The repo gives a Diffusers pipeline, a Gradio app, a ComfyUI plugin, LoRA training guidance, a prompt-enhancer skill and day-0 SGLang and vLLM-Omni support.

github.comrepoAdded 20 Oct 2026, published 16 Oct 2026🧠 AI

Announces Reflection's Beam-1.0-501B-A23B, an Apache 2.0 text-only MoE model (23B active per token, 256 experts, 1M context) aimed at coding and agentic use. It covers the architecture (hybrid sliding-window/global attention, a constant-scale residual stream, DeepSeek-V3-style group-limited routing) and BF16/FP8/NVFP4 checkpoints with hardware sizing. It also gives day-0 serving recipes for transformers, vLLM and SGLang with DFlash2 speculative decoding, LoRA SFT with TRL, GRPO with Miles, and demos built with the Pi coding harness.

huggingface.coarticlePublished and added 16 Oct 2026🧠 AI

Hugging Face's launch post for Meta's Muse Spark Open, an open-weight 816B-parameter mixture-of-experts model (about 42B active per token). It has 256 experts with 8 routed per token, a 1M-token context, and perception and audio encoders for image, video and audio input. The post gives the architecture details, BF16 and NVFP4 checkpoints, and run recipes for transformers, llama.cpp, SGLang, vLLM, MLX and Inference Endpoints. It also covers LoRA SFT of the full model with TRL on 64 H100s and agentic demos (Pi 3D asset generation, an OpenEnv browser agent, a voice-driven shopping agent).

huggingface.coarticlePublished and added 15 Oct 2026🧠 AI

Liquid AI releases LFM2.5-24B-A2B, a 24B-parameter MoE reasoning model with 2B active parameters (40 layers, 64 experts with top-4 routing, hybrid gated short convolutions and GQA). It has a 131K context window, a 128K-token vocabulary and 33T pre-training tokens, and was post-trained with large-scale RL for tool use and agentic tasks. The post reports results against Qwen3.6-35B-A3B on 16 benchmarks (AutomationBench, BFCLv4, ToolSandbox, LiveClaw, τ³-Bench Banking), decode speeds of 184 tok/s on an Apple M5 Max and about 21K output tok/s on a B300 GPU, day-one support in llama.cpp, MLX, vLLM, SGLang and ONNX, and a local Hermes Agent demo. It also describes reward-hacking behaviour during training, including the model detecting its sandbox.

liquid.aiarticleAdded 16 Oct 2026, published 14 Oct 2026🧠 AI

Adversa AI's October 2026 roundup of 38 GenAI security resources, grouped into attack techniques, defenses, red teaming, vulnerabilities, incidents, articles and frameworks. Highlights include Anthropic's finding that GLM-5.3 safeguards collapse under prefill and abliteration, capability laundering and activation-dilution jailbreaks, RAG poisoning attacks, the LiteLLM proxy flaw (CVE-2026-35029), and the SGLang pickle RCE (CVE-2026-86793). It also covers a joint US advisory on industrial-scale model distillation. It closes with advice to assume model safeguards can be removed and to harden the serving stack around the model.

adversa.aiarticlePublished and added 13 Oct 2026🛡️ Security

AINews issue for 10/8–10/9/2026 from Latent Space. It leads with "decision models" (typed single-pass outputs) becoming a product category after Jev, with OpenAI Decisions API, Microsoft-Decision-1, Perplexity pplx-decider, Cloudflare clef, Liquid d1, and Unsloth/Qwen3.5 training recipes. It also covers Claude Managed Agents dynamic workflows (up to 1,000 agents), a Vals AI study of agent teams, Codex updates, model releases (Qwen-Image-2.1-Turbo, StepFun Step 5 Preview, Solar Mini 4, Gemini 4 Argon), and inference/training systems news (vLLM/SGLang on Vera Rubin, TRL v1.15, DeepSeek stride weak spots).

latent.spacearticlePublished and added 10 Oct 2026🧠 AI

Model card for Qwen3.8-27B, a 27B dense vision-language model from the Qwen team with a hybrid Gated DeltaNet / Gated Attention layout (64 layers, 16×(3 DeltaNet + 1 attention)), MTP training, and 262k native context extensible to 1M via YaRN. It targets coding, research and long-horizon agentic tasks, with thinking on by default, reasoning_effort levels (xhigh/medium/low) and preserve_thinking. The card gives deployment recipes for Transformers, vLLM, SGLang and TokenSpeed, recommended sampling parameters, video-token settings for hour-scale video, and benchmark notes (SWE-bench Pro, NL2Repo-Bench, DeepSWE, HLE, MathVision, WebArena-Verified, etc.).

huggingface.corepoAdded 9 Oct 2026🧠 AI

Hy4 preview is Tencent's new flagship Mixture-of-Experts language model: 770B total parameters with 49B activated per token across 78 layers (256 routed experts + 1 shared expert, top-8 routing), plus a native 10B multi-token-prediction layer for speculative decoding. It introduces Gated DeepSeek Sparse Attention with IndexCache for cross-layer index reuse and identity Hyper-Connections (iHC) on the residual path, and in blind internal evaluations on engineering tasks it edges out GLM-5.3 and Kimi K3. Weights (including an FP8 variant) are released under Apache 2.0 on Hugging Face with vLLM/SGLang deployment recipes and AngelSlim-based quantization/compression tooling.

huggingface.corepoAdded 2 Oct 2026🧠 AI

Announcement of an updated model card for JEV-27B-VL, an open-weight vision-language model released under Apache-2.0. The card covers the model's weights and a "decision adapter," a BoE (Bag/Blend of Experts) architecture paired with a System 1 / System 2 reasoning setup, calibration details and decision prompts, serving instructions for vLLM and SGLang, and an evaluation of image recommendation quality.

x.comthreadPublished and added 2 Oct 2026🧠 AI

DRelay is a speculative decoding technique for LLM inference that repairs parallel draft-token selections before target-model verification, using a global reader over the entire draft block plus a causal selector to decide whether to keep or replace each candidate token, extending the accepted prefix length. Trained jointly with the draft backbone via a combined candidate-support and position-weighted repair loss, it is evaluated on eight benchmarks on an H800 GPU and, under SGLang serving, improves end-to-end decoding speedup substantially over DFlash, Domino, and DSpark baselines.

arxiv.orgpaperAdded 2 Oct 2026, published 1 Oct 2026🧠 AI

DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters, 1M-token context, and a novel Causal Encoder-Decoder (CED) architecture where the decoder's global KV cache is projected from encoder hidden states rather than recomputed per layer, activating only 8B/16B parameters per token during prefill/decode. It introduces Compressed Sparse Attention 2 (CSA2) with FP4 KV caching to cut the global KV cache footprint to 890 bytes/token (~1/4 of DeepSeek-V4-Flash, ~1/437 of DeepSeek-V1), alongside Engram conditional memory, DSpark speculative decoding, and a continuously controllable reasoning-effort setting (1–100). The card documents pre/post-training recipes (45T tokens, SFT→RL→on-policy distillation with synthetic agent environments) and reports results across agentic and code benchmarks (Terminal-Bench 2.1/3.0/4.0, DeepSWE v1.1, GPQA Diamond, Chartography, ZeroBench, AutomationBench), plus deployment instructions for Transformers, vLLM, SGLang, and Docker.

huggingface.corepoAdded 30 Sep 2026🧠 AI

STEPQuant is a spatial-temporal post-training quantization framework for the fixed-size recurrent states used by Delta-rule linear attention, which become a memory bottleneck under concurrent serving. It allocates bit precision based on error magnitude and memory lifetime (temporal) and fits per key-row/value-column scales based on state distribution and output impact (spatial), achieving near-FP32 accuracy at a 6-bit budget and beating uniform INT8 at 4-bit. Evaluated on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct, and integrated into SGLang with custom GPU kernels, it delivers over 5x recurrent-state compression and up to 68.7% total serving memory reduction.

arxiv.orgpaperAdded 30 Sep 2026, published 29 Sep 2026🧠 AI

SemiAnalysis article on GLM-5.3 and sparse attention. It explains why DeepSeek Sparse Attention does not cut KV cache capacity (top-k needs full context in HBM) and how SGLang's HiSparse offloads KV to host DRAM. It covers InferenceX AgentX serving cost on GB200, GB300 and MI355X (about $0.044 vs $0.049 per million tokens at 150 tok/s), IndexShare, and GLM's RL post-training (SAO, slime).

newsletter.semianalysis.comarticleAdded 29 Oct 2026, published 28 Sep 2026🧠 AI

MiMo-V2.6-Pro-RL is Xiaomi's flagship omnimodal (text/image/video/audio) sparse MoE model, 1.02T total / 42B activated parameters with a 1M-token context, built around scaling RL compute toward self-improvement. It uses a single mixed GRPO run across coding, agentic, visual and cybersecurity tasks, a novel groupwise agentic grading scheme (Groupwise Reward Synthesis + Groupwise Advantage Redistribution) to rank passing rollouts rather than just pass/fail, and a multi-teacher on-policy distillation stage (MOPD2); it also ships a 5-layer speculative MTP decoder and is deployable via vLLM and SGLang.

huggingface.corepoAdded 28 Sep 2026🧠 AI

MiMo-V2.6-Flash-RL is Xiaomi's efficiency-balanced checkpoint in the MiMo-V2.6 series, a 309B-total/15B-activated sparse MoE model with native text, image, video and audio support and a 1M-token context window. It's trained with a single mixed reinforcement learning run (GRPO at large batch scale) across coding, agentic, visual and cybersecurity tasks, using a novel Groupwise Agentic Grading scheme (Groupwise Reward Synthesis and Groupwise Advantage Redistribution) to build a self-improvement loop, followed by Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2). The card documents the architecture (MiMo ViT vision encoder, AudioTokenizer audio encoder, 5-layer speculative MTP decoder) and deployment recipes for vLLM and SGLang.

huggingface.corepoAdded 26 Sep 2026🧠 AI

Ling-3.0-flash-VL is InclusionAI/Ant Group's native multimodal model built on Ling-3.0-flash, adding image and video understanding via a ViT encoder, VideoRoPE, and a hybrid MoE backbone (42 layers alternating KDA and Gated MLA at 5:1) with 124B total parameters but only 5.5B activated per token, and a 256K context window. It targets agentic multimodal workflows — understanding, reasoning, and acting on visual interfaces — scoring 42 on Artificial Analysis Intelligence Index v4.1.1, up from 38 for the text-only Ling-3.0-flash. The card includes concrete SGLang and vLLM serving recipes for various GPU configurations.

huggingface.corepoAdded 26 Sep 2026🧠 AI

Liquid AI releases LFM2.5-VL-3B-DSpark, a 280M-parameter attention-only draft model (4 layers, block size 9, +8.9% parameters) for speculative decoding of the LFM2.5-VL-3B vision-language model. It reports decode speedups of 2.30–3.13x on MLX (M5 Max), 1.57–2.14x on llama.cpp (M3 Ultra) and up to 2.66x on H100, with end-to-end gains of 1.30–2.62x, evaluated on six MMSpec vision tasks. It explains why prefill and vision encoding limit end-to-end gains (Amdahl's law) and gives launch commands for SGLang, llama.cpp and MLX-VLM.

huggingface.coarticleAdded 26 Sep 2026, published 24 Sep 2026🧠 AI

SemiAnalysis reports first agentic inference results for NVIDIA Vera Rubin NVL72 on its AgentX benchmark (InferenceX), using DeepSeek V4 Pro on early TensorRT-LLM software. At 170 TPS it claims about 67x GB300 throughput per TCO, and 2.09x GB300 SGLang tokens per MW at 100 TPS. It also models revenue and profit per gigawatt, and notes SGLang narrows the GB300 gap.

newsletter.semianalysis.comarticleAdded 29 Oct 2026, published 14 Sep 2026🧠 AI

Reef is an open-source infrastructure stack for continual self-improving agents, wiring together agent inference, scored feedback, training/updates, and versioned delivery into one loop. It supports two learning surfaces: fine-tuning model weights via RL training steps (using Slime and SGLang on top of models like Qwen2.5-1.5B-Instruct) or evolving an agent's harness — prompts, rules, and skills — through its built-in "Reefine" recipe, which lets a model propose and install harness changes from plain-language requests. It exposes an OpenAI/Anthropic-compatible inference endpoint plus a reporting endpoint for scoring interactions, and ships a recipe catalog (e.g. the SAO recipe referencing arXiv:2607.07508) for pluggable learning strategies.

github.comrepoAdded 2 Oct 2026, published 31 Aug 2026🧠 AI

LoongSage is Baidu's open-sourced, production-grade reinforcement learning framework for post-training LLMs, built on Ray for scheduling, Megatron-Core/Megatron-Bridge for training, and SGLang for inference. It targets agentic RL with a zero-intrusion AgentFlow module (automatic multi-turn context, loss masking, trajectory collection), multi-paradigm online distillation (PG-Style, GKD-Style, TopK/full-vocab/JSD KL), fully asynchronous training with partial rollout, and system-level training-inference consistency guarantees (Router Replay, TITO, M2PO, OPSM) that keep probability bias at 1e-4 in MoE/low-precision settings. Recipes are validated on Baidu's internal production runs of DeepSeek-V4-Flash and Qwen3-30B-A3B for math reasoning, agentic code repair, and distillation.

github.comrepoAdded 30 Sep 2026, published 28 Aug 2026🧠 AI

Datawhale and RadixArk (the SGLang team's company) run this hands-on course on LLM inference. It starts with KV Cache, prefill vs. decode, and compute-bound vs. memory-bound. It then builds a mini-sglang step by step: forward pass, generation, HTTP serving, Continuous Batching, Paged KV Cache and RadixAttention. The later parts cover the real SGLang source, including quantization, hierarchical caching, DP Attention/EP/PP and Prefill-Decode disaggregation, plus profiling and the PR workflow. It is a rare code-first tutorial on inference engines, available in English and Chinese. Part 0, Part I and Part II chapters 1–2 are done, and the rest is in development.

github.comcourseAdded 12 Oct 2026, published 27 Aug 2026🧠 AI

Halo is White Circle's open-source training framework that runs the same codebase from a single GPU to multi-node clusters, adding Expert/Context/Tensor/Expert-Tensor parallelism (EP/CP/TP/ETP) directly onto native Hugging Face models via FSDP2, DTensor and DeviceMesh rather than a separate distributed model implementation. It covers the full training lifecycle — pre-training, SFT, preference optimization, distillation, and fully asynchronous multi-turn RL with a Transformers-side trainer pushing weights over NCCL to vLLM/SGLang rollout servers — and reports up to 2.8x the throughput of stock TRL for gpt-oss-20b on 8x B300 GPUs. It ships as prebuilt Docker images for Blackwell and Hopper GPUs with FlashAttention-4, DeepEP V2, DeepGEMM, and a bf16 AdamW optimizer integrated, supports 15 MoE families, and includes LoRA/QLoRA paths that fit consumer GPUs.

github.comrepoAdded 30 Sep 2026, published 26 Aug 2026🧠 AI

IBM's technical write-up on Granite 4.2, its first family of dense decoder-only reasoning LLMs (3B, 8B, 30B), post-trained from Granite-4.1 base models pretrained on ~15T tokens with a five-phase schedule extending context to 512K tokens. It details the SFT data mixture (agentic vs. non-agentic, quality filtering with LLM judges GPT-OSS-120B and Gemma 4), a staged asynchronous-GRPO RL pipeline (RLVR, skill boosters, then agentic SWE/Terminal/Search stages for 8B/30B, finishing with RLHF), and the NeMo-RL/NeMo-Gym infrastructure used to run tool-using RL rollouts at scale, plus FP8/NVFP4/MXFP4/GGUF quantized releases and integration guides for vLLM, SGLang, OpenCode, Pi, and OpenHands.

huggingface.coarticleAdded 29 Sep 2026, published 25 Aug 2026🧠 AI

Liquid AI describes DSpark, a speculative-decoding method combining a DFlash-style parallel draft backbone, a lightweight sequential (Markov-chain) head, and a confidence-scheduled verifier that prunes low-confidence draft suffixes. They release ~300M-parameter DSpark draft models for LFM2.5-2.6B, LFM2.5-1.2B-Instruct and LFM2.5-8B-A1B (Safetensors and GGUF), reporting up to 3.18x GPU throughput (SGLang on H100) and up to 2.87x on-device throughput (llama.cpp/Metal on M4 Max), plus a 57% average cut in function-calling latency, with day-one upstream support in llama.cpp and SGLang.

huggingface.coarticleAdded 29 Sep 2026, published 20 Aug 2026🧠 AI

A 10-week, 30-minutes-a-day roadmap (GitHub repo "time-to-first-token") that walks engineers through building one continuously-evolving OpenAI-compatible inference service rather than disconnected tutorials: standing up vLLM and SGLang, reading the PagedAttention/RadixAttention source, instrumenting Grafana dashboards for TTFT/inter-token latency/throughput/queue depth, load-testing past 1000 concurrent requests with GuideLLM, benchmarking FP8/INT4/AWQ quantization and speculative decoding, configuring KV cache eviction, deploying prefill/decode disaggregation on Kubernetes, and building a cost/latency/quality router. It closes with a checklist of common public-benchmark mistakes (sweep request rate, report p50/p95/p99, separate TTFT from inter-token latency, pin versions) and pointers to Stanford CS336, MIT EfficientML.ai, vLLM Office Hours, and GPU MODE for further study.

github.comrepoAdded 2 Oct 2026, published 2 Aug 2026🧠 AI