vLLM
Named by 105 kept items in this workspace.
96 items name it under frameworks and tools and 9 under classification.technologies.
The items that name it 105
Serving engine; supports Mistral Large 4 FP8 and NVFP4 checkpoints with a built-in DSpark drafter for 1.92 to 2.23x output throughput.
Serving engine with day-0 support for Nemotron 3.5 Transcribe.
Serving engine with day-0 support; serves the NVFP4 checkpoint (needs NVIDIA Blackwell GPUs) and uses the DSpark drafter for speed-ups.
Inference engine named among fast-growing community and independent projects.
Serves the Nemotron 3.5 Transcribe checkpoint.
Inference server; day-one support for MiniCPM-OCR 1.0, and used in NVIDIA Rubin throughput comparison.
Inference engine with native LFM3 support; used as the comparison serving setup.
Inference serving engine used on SemiAnalysis AgentX for DeepSeek V4.1-Flash.
Serving engine with paged KV caches; unmodified vLLM serves folded caches through a KV connector.
Inference engine used to serve open-weight models. Added constrained decoding for tool calls in v0.25, turned it off by default, and added the --tool-strict-level flag. Version 0.31 released it as a server-side flag.
Serving framework run on AMD chips to serve Kimi K3, which the page says can earn up to $3/hr/chip in profit.
Serving framework; Nanbeige's example splits Nanbeige4.3-33B across 4 GPUs.
Inference framework. A year ago open models were released first and vLLM patched afterwards. Now day-one support is typical.
Widely used GPU LLM serving system, used as a baseline.
Inference engine trusted by the ML community for day 0 open model support.
Serving engine with day-0 support for MiniMax M3.1 Flash and Sage-A1.5.
Serves the merged fine-tuned checkpoint for evaluation, with qwen3 reasoning and tool-call parsers.
Serving framework; vLLM-TorchTPU adds prefix caching, speculative decoding and disaggregated serving.
Inference server used to run the 500 test requests on an RTX 5090 GPU for the grader cost measurements.
Inference serving engine. With Wide-EP and EPLB it matches DeepInfer's decode performance within 6%, per the post.
Supported serving runtime for granite-speech-5.0-2b.
Open-source serving engine on which DeepInfer was initially built; the vLLM team reported decode throughput within 6% of DeepInfer on GB200 under a matched configuration.
Open-source serving engine; DeepInfer builds on it. vLLM reported decode throughput within 6% of DeepInfer on GB200 and published AgentX results and recipes.
Serving project whose team worked with Google on TorchTPU compatibility.
Existing open-source inference tool, compared with DeepInfer.
DeepInfer was originally built primarily on vLLM.
Serving engine; v0.32.0 (814 commits) adds a /v1/systemone endpoint, low-precision KV including NVFP4 and 4-bit KV on ROCm, a GPU n-gram speculator and multi-API workers.
Self-hosted OpenAI-compatible model server that Antirion can front. Pool scheduling reads its load, prefix and LoRA adapter state.
Serves the model with an OpenAI-compatible API (vllm serve)
Serving engine supporting Beam at launch
Inference engine; day-0 support for Muse Spark Open and Reflection Beam, used for RL rollouts in Muse training.
Inference serving engine; day-0 support for Muse Spark Open and Reflection Beam; used for RL rollouts in Muse training
Serving engine that Muse Glimmer works with
Inference engine used to serve NVIDIA's FP4-Local checkpoint (via eugr's b12x container)
Serving engine with day-0 Muse Spark Open support on NVIDIA, AMD MI355X and TPU; bitwise-consistent RL; rollouts for LoRA GRPO
Day-0 serving of Beam with an OpenAI-compatible API, tensor parallelism and DFlash speculative decoding
Serving engine; IBM's team contributed to speed up low rank adapter inferencing.
Serving engine; day-0 support for Ling-3.1-flash; adaptive verification for speculative decoding with headless MTP, EAGLE, DFlash and DSpark drafters without retraining.
Serving Muse Spark Open NVFP4 (text and image) on Blackwell GPUs via a reference plugin with reasoning and tool-call parsers.
GPU-accelerated serving for production throughput; used for evaluations
Inference engine; added ReplaySSM-backed Mamba2 decoding/MTP support
LLM serving framework that a user wants used with Qwen3.8-27B on DGX Spark.
Inference backend with day-one support; used for benchmark evaluation; port 8000
Inference serving engine used for the Gemma 4 E2B speedup
Serving engine; AMD megakernel decode landed upstream on ROCm
Serving; >7.8x GB200 throughput on MiniMax M3 on Vera Rubin (early); locality-aware MoE
LLM serving framework in which ReplaySSM is implemented; tuned vLLM baseline and vLLM speculative decoding used for comparison
Serving engine with day-0 support for EmbeddingGemma 2
Used to serve the model efficiently
Serving backend to which the multi-harness RL capture proxy forwards calls.
Serving engine used to run Kolibri; Aleph Alpha's plugin supports one specific vLLM version at a time (0.29 at launch).
Serving engine used by Prime Inference to serve GLM-5.3
Serving instructions provided for the model
LLM inference/serving tool cited as part of what drove renewed interest in GPU kernel writing.
Inference engine that Matt Mastracci contributed pull requests to, to make DiffusionGemma support stronger.
Inference serving engine; added day-0 support for IQuest-Q1
Serves the models, one server per GPU with the decision server in front. The package carries a vLLM plugin that Instant models need. Version 0.31.0 is used; published numbers were measured on 0.19.1, which is also supported.
Inference serving engine used with TileRT to reach 469 tok/s decode for GLM-5.3 on AMD MI355X
Serving engine. Keeping the first decoding step on a consistent CUDA graph path cut TPOT from about 40 ms to 22 ms. Implements MHA mode for MLA at certain sequence lengths and handles prefill for TileRT.
Standard LLM serving system used as the baseline that DeepEdu's deployed configuration is compared against for TTFT speedup
inference serving engine, used with TileRT to reach high decode throughput on GLM-5.3
Listed as a supported vendor.
Used to time package installs and vllm serve start-up until /v1/models responds.
inference serving engine; published tuned serving configs for Qwen3.8-2.4T on GB300 NVL72 and integrated PyNvVideoCodec/NVDEC to remove CPU decode bottlenecks in video workloads
Inference server that works with the orchestration layer and has its own orchestration features.
Implements PagedAttention by default for KV cache memory management and uses continuous batching as its default scheduling strategy in production.
Inference library accompanying open-weight model releases
Example of an OpenAI-compatible provider that can be used as the LLM upstream
LLM serving engine; used to serve DeepSeek-V4.1-Flash. SemiAnalysis built an unoptimized fork that stores Engram tables in memory-mapped SSD files.
LLM serving engine with PagedAttention forked by the typesafe-ai GitHub org; paged KV cache plus prefix caching makes extra questions over one shared state nearly free.
Serving Index-Translate models (OpenAI-compatible server); FP8 and NVFP4 quantized builds target it
- [AINews] AEF-1 standard emerges for Third Party Evaluators, as Xai, OpenAI, and Anthropic all cosign
Serving engine gaining first-class TPU support via a partnership between Inferact and Google Cloud, including production serving features and optimized kernels
Serving stack used for B200 and B300 comparisons; Rubin vLLM results are expected in later articles.
Inference server used to serve the base model plus dynamically loaded LoRA adapters via /v1/load_lora_adapter; keeps several adapters loaded at once and stores a block-based KV prefix cache
Open-source production inference serving engine; its TPU support went from PyTorch/XLA to tpu-inference (JAX/TorchAX) to TorchTPU.
Rollout/serving engine behind online and environmental RL; trainer pushes updated weights to it over native NCCL.
Used to serve Granite 4.2 through an OpenAI-compatible endpoint, emitting tool calls in the OpenAI function-calling format so it plugs into agentic harnesses without extra glue; also generates rollouts on the training side (NeMo-RL) and is the backend for quantized inference.
Mentioned as an alternative that could be used to serve the embedding model instead of TEI
Local serving option usable as an OpenAI-compatible endpoint for the AliangBoard agent
Running open-weight models on your own machines
OpenAI-compatible inference serving engine; implements PagedAttention and continuous batching, with chunked prefill as its default scheduling strategy
Can be run locally and pointed to via OPENAI_BASE_URL as an alternative model-serving provider.
Inference serving framework; patches for new open model releases are now often ready days or weeks before launch, part of the open ecosystem's professionalization.
Serving framework; used to disable Kimi K2.5's reasoning via chat_template_kwargs={"thinking": False}, and referenced as a self-hosting option for GLM-5's chat template.
Single-engine inference operation target that the LiteLLM skill routes to for engine-specific concerns
Supports running local models with Sim.
Inference server (vllm serve) exposing an OpenAI-compatible API for the model
Serving engine with OpenAI-compatible API; supports YaRN and video fps sampling
Used to serve Kolibri with reasoning and tool-calling parsers enabled, supports contexts up to 1,048,576 tokens with overrides
Recommended production serving engine for Hy4 preview; supports serving via OpenAI-compatible API, tensor parallelism, speculative decoding (MTP), and the FLASHMLA_SPARSE attention backend
Serving engine to host DeepSeek-V4.1-Flash and expose an OpenAI-compatible API via 'vllm serve'
Serving library used to install and serve the model with an OpenAI-compatible chat completions API
LLM inference and serving engine; subject of this v0.30.0 release covering models, kernels, quantization, and serving infrastructure
Serve MiMo-V2.6-Pro-RL with an OpenAI-compatible API via 'vllm serve'
Serving engine to host an OpenAI-compatible API server for MiMo-V2.6-Flash-RL
Inference serving framework used to run Ling-3.0-flash-VL via a fork (vllm-ling-v3), served with tensor parallelism, prefix caching, and tool/reasoning parsers
In every workspace you can read 126
AI
Security
All technologies in this workspace.