30 results for “vLLM”

Antirion is an open-source AI gateway shipped as one Rust binary with a built-in admin console. It sits between apps and providers such as OpenAI, Anthropic, Bedrock, Vertex and vLLM, translating between OpenAI, Anthropic, Gemini and Bedrock dialects. It covers routing with fallbacks and circuit breakers, budgets, guardrails, caching, MCP and A2A gateways, SSO, and a benchmark against Bifrost 2.2.4 and LiteLLM 1.103.1.

github.comrepoAdded 21 Oct 2026, published 17 Oct 2026🧠 AI

Model card for DeepSeek-V4.1-Flash-1017, a minor update to DeepSeek-V4.1-Flash with MIT-licensed weights. It claims gains in reasoning, coding, agentic tasks and visual benchmarks, with shorter task time and output length that move the performance–cost Pareto frontier across agent harnesses (Claude Code for Terminal-Bench 4.0, Mini-SWE-V2 for DeepSWE v1.1). It ships no Jinja chat template. It provides a reference encoding.py for multi-turn chat, tool calling, thinking mode, numeric reasoning effort and interleaved images, plus deepseek-recipe, Rust libraries with Python bindings for prompt encoding and output parsing. Serving recipes are given for Transformers, vLLM, SGLang and Docker Model Runner.

huggingface.corepoAdded 18 Oct 2026, published 17 Oct 2026🧠 AI

Model card for Meta Superintelligence Labs' Muse Spark Open, an open-weights, natively multimodal model with 816B total and 42B active parameters and a 1M-token context window. It targets coding and agentic work (tool use, long-horizon tasks, multimodal reasoning, synthetic data generation, LLM-as-judge). It ships in BF16 and NVFP4, needs about 700GB VRAM (NVFP4) or 2TB (BF16), runs on SGLang, vLLM, Transformers, TensorRT-LLM, Unsloth, llama.cpp and MLX, and supports a reasoning_effort setting. The Muse Model License requires "Built with Muse Spark Open" attribution and a separate agreement for Model-as-a-Service above $20M revenue. The card reports safety results against Kimi K3 on cyber, chem/bio and loss-of-control evals.

huggingface.corepoAdded 17 Oct 2026🧠 AI

MOSS-Image is an open-source text-to-image model family from OpenMOSS with a 5B backbone, released October 2026 as MOSS-Image-Base and MOSS-Image-SFT. It supports JSON layout control with bounding boxes, text rendering inside images, and prompts up to 4K tokens. The repo gives a Diffusers pipeline, a Gradio app, a ComfyUI plugin, LoRA training guidance, a prompt-enhancer skill and day-0 SGLang and vLLM-Omni support.

github.comrepoAdded 20 Oct 2026, published 16 Oct 2026🧠 AI

Model card for Qwen3.8-27B, a 27B dense vision-language model from the Qwen team with a hybrid Gated DeltaNet / Gated Attention layout (64 layers, 16×(3 DeltaNet + 1 attention)), MTP training, and 262k native context extensible to 1M via YaRN. It targets coding, research and long-horizon agentic tasks, with thinking on by default, reasoning_effort levels (xhigh/medium/low) and preserve_thinking. The card gives deployment recipes for Transformers, vLLM, SGLang and TokenSpeed, recommended sampling parameters, video-token settings for hour-scale video, and benchmark notes (SWE-bench Pro, NL2Repo-Bench, DeepSWE, HLE, MathVision, WebArena-Verified, etc.).

huggingface.corepoAdded 9 Oct 2026🧠 AI

Kolibri-1 is Aleph Alpha's 78B-parameter mixture-of-experts reasoning model focused specifically on German and English, released under Apache 2.0 with weights, a vLLM plugin, and an extensive model card. It supports explicit reasoning-effort levels, Hermes-style tool calling, and a native 262144-token context validated to over 1M via sliding-window attention; the card details its UniBPE German-aware tokenizer, pre-training mid-training and long-context training recipe, MergeMix-based data-mix search, Merlin-Arthur hallucination-mitigation RL protocol, and benchmark results against Qwen3.5, Gemma 4, Nemotron 3 Nano, and Apertus on MMLU-Pro, GPQA, AIME, SWE-bench Verified, Terminal-Bench 2.1, and HLE.

huggingface.corepoAdded 8 Oct 2026🧠 AI

Overmind is an open-source platform (MIT SDK/CLI, AGPL-3.0 platform, self-hostable) that instruments an agent's code via OpenTelemetry tracing, turns the resulting production traces (or any dataset) into a fine-tuned model benchmarked against user-defined eval metrics, and serves it through one unified API with downloadable/retrainable weights. It ships an SDK, CLI, REST API, Console, and an MCP server exposing project tools and workflows (investigate-capability, finetune-capability, ship-model) for Cursor, Claude Code, OpenCode and Codex, and can sync traces from Langfuse, LangSmith, Braintrust or Galileo; self-hosting runs via Docker Compose with Postgres, Redis, Celery workers and Modal-based GPU workers for fine-tuning/serving (vLLM).

github.comrepoAdded 6 Oct 2026🧠 AI

Hy4 preview is Tencent's new flagship Mixture-of-Experts language model: 770B total parameters with 49B activated per token across 78 layers (256 routed experts + 1 shared expert, top-8 routing), plus a native 10B multi-token-prediction layer for speculative decoding. It introduces Gated DeepSeek Sparse Attention with IndexCache for cross-layer index reuse and identity Hyper-Connections (iHC) on the residual path, and in blind internal evaluations on engineering tasks it edges out GLM-5.3 and Kimi K3. Weights (including an FP8 variant) are released under Apache 2.0 on Hugging Face with vLLM/SGLang deployment recipes and AngelSlim-based quantization/compression tooling.

huggingface.corepoAdded 2 Oct 2026🧠 AI

AI-Infra-Guard (A.I.G) is Tencent Zhuque Lab's open-source AI red-teaming platform combining AI infrastructure vulnerability scanning (2000+ CVE rules covering vLLM, llama.cpp, LangFlow, n8n, MLflow and 145+ other AI components), MCP server/skill security scanning, LLM API poisoning detection, and multi-turn jailbreak evaluation (Many-Shot, PAIR, GOAT, ActorAttack). It ships CLI tools (skill-scan, mcp-scan, agent-scan), a Docker-deployed web UI, and has produced original research including FORGE-Bench (loss-of-control benchmark), RogueHandoff-20, and SkillJack (persistent skill backdoors), with talks at Black Hat Europe 2025 on MCP security.

github.comrepoAdded 30 Sep 2026🛡️ Security

DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters, 1M-token context, and a novel Causal Encoder-Decoder (CED) architecture where the decoder's global KV cache is projected from encoder hidden states rather than recomputed per layer, activating only 8B/16B parameters per token during prefill/decode. It introduces Compressed Sparse Attention 2 (CSA2) with FP4 KV caching to cut the global KV cache footprint to 890 bytes/token (~1/4 of DeepSeek-V4-Flash, ~1/437 of DeepSeek-V1), alongside Engram conditional memory, DSpark speculative decoding, and a continuously controllable reasoning-effort setting (1–100). The card documents pre/post-training recipes (45T tokens, SFT→RL→on-policy distillation with synthetic agent environments) and reports results across agentic and code benchmarks (Terminal-Bench 2.1/3.0/4.0, DeepSWE v1.1, GPQA Diamond, Chartography, ZeroBench, AutomationBench), plus deployment instructions for Transformers, vLLM, SGLang, and Docker.

huggingface.corepoAdded 30 Sep 2026🧠 AI

StartLux-Decision is a family of open decision models (0.8B to 27B, plus a 35B-A3B mixture of experts) that return probabilities for every option of typed choice, yes/no and score questions. The repository holds the inference engine and HTTP server (vLLM plugin, /v1/systemone API), with Instant one-pass and AutoThink gated-reasoning variants. Reports first place of 174 entries on Decision Index 0.3.2 (69.12). Code is Apache 2.0; weights are CC BY-NC 4.0.

github.comrepoAdded 23 Oct 2026, published 29 Sep 2026🧠 AI

Release notes for vLLM v0.30.0, a major update to the open-source LLM inference/serving engine spanning 762 commits. Highlights include new model support (DeepSeek-V4.1-Flash, GLM-5.3-Flash, Kimi K3, Qwen3.8-Flash-Next, K2-Horizon, Cohere Compass and others), a "Fast Start" GPU weight-cache daemon for near-instant engine restarts via CUDA IPC, Gumbel-max watermarking for generated text, a HiSparse host-memory tier for sparse-MLA KV offload, Model Runner V2 improvements (dual-batch overlap, speculative decoding under pipeline parallelism), expanded quantization support (NVFP4, W4A16 DSA, AutoRound), and hardware-specific tuning across NVIDIA, AMD ROCm, Intel XPU and CPU backends.

github.comrepoAdded 28 Sep 2026🧠 AI

MiMo-V2.6-Pro-RL is Xiaomi's flagship omnimodal (text/image/video/audio) sparse MoE model, 1.02T total / 42B activated parameters with a 1M-token context, built around scaling RL compute toward self-improvement. It uses a single mixed GRPO run across coding, agentic, visual and cybersecurity tasks, a novel groupwise agentic grading scheme (Groupwise Reward Synthesis + Groupwise Advantage Redistribution) to rank passing rollouts rather than just pass/fail, and a multi-teacher on-policy distillation stage (MOPD2); it also ships a 5-layer speculative MTP decoder and is deployable via vLLM and SGLang.

huggingface.corepoAdded 28 Sep 2026🧠 AI

MiMo-V2.6-Flash-RL is Xiaomi's efficiency-balanced checkpoint in the MiMo-V2.6 series, a 309B-total/15B-activated sparse MoE model with native text, image, video and audio support and a 1M-token context window. It's trained with a single mixed reinforcement learning run (GRPO at large batch scale) across coding, agentic, visual and cybersecurity tasks, using a novel Groupwise Agentic Grading scheme (Groupwise Reward Synthesis and Groupwise Advantage Redistribution) to build a self-improvement loop, followed by Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2). The card documents the architecture (MiMo ViT vision encoder, AudioTokenizer audio encoder, 5-layer speculative MTP decoder) and deployment recipes for vLLM and SGLang.

huggingface.corepoAdded 26 Sep 2026🧠 AI

Ling-3.0-flash-VL is InclusionAI/Ant Group's native multimodal model built on Ling-3.0-flash, adding image and video understanding via a ViT encoder, VideoRoPE, and a hybrid MoE backbone (42 layers alternating KDA and Gated MLA at 5:1) with 124B total parameters but only 5.5B activated per token, and a 256K context window. It targets agentic multimodal workflows — understanding, reasoning, and acting on visual interfaces — scoring 42 on Artificial Analysis Intelligence Index v4.1.1, up from 38 for the text-only Ling-3.0-flash. The card includes concrete SGLang and vLLM serving recipes for various GPU configurations.

huggingface.corepoAdded 26 Sep 2026🧠 AI

An extensive, actively maintained GitHub "awesome list" cataloging the ecosystem that has grown around TypeSafe's Jev and its "System One" typed decision models — a non-generative alternative to chat LLMs that returns calibrated probabilities per option (Choice/Score/Noul primitives) via RLCD training and single-pass constrained decoding. It indexes dozens of open-weight reproductions (OneJev, Rizzo Flow, Laya, Jebadiah, kev), benchmarks and leaderboards (JevBench, Jev Decision Index, JevAdvBench, DecisionBench), downstream tools (browser/robot/trading agents built on the typed API), and the research lineage the design draws on (zero-shot entailment classifiers, InstructGPT's reward model, GLiNER/GLiClass, monoBERT, LLaDA, vLLM, DSPy, RouteLLM, Constitutional Classifiers). It also links commentary auditing TypeSafe's claims (accuracy vs. frontier LLM baselines, security issues found in community tools).

github.comrepoAdded 29 Sep 2026, published 17 Sep 2026🧠 AI

Decider is an open reproduction of TypeSafe AI's System One (Jev) model class: instead of generating text, it reads a state plus a set of typed questions (Choice, Score, or Noul/yes-probability) and returns a calibrated probability distribution for each in one forward pass, with no decoding or parsing. The repo ships 2B/4B/12B and a 35B mixture-of-experts checkpoint (built on Qwen3.5 and Gemma-4 base models), trained on roughly 95 public datasets plus teacher-labelled data, with per-type temperature calibration, GGUF/llama.cpp and vLLM serving paths, an MPS/Apple Silicon port, and a community Snapdragon X Elite NPU port. It benchmarks against TypeSafe's Jev on JevBench and the Decision Index, documenting where it closes the gap (language, tools, retrieval) and where it lags (knowledge-heavy MMLU/GSM8K/GPQA tasks and calibration on hard items).

github.comrepoAdded 2 Oct 2026, published 16 Sep 2026🧠 AI

Index-Translate is Bilibili's Apache-2.0 family of multilingual, multimodal translation models built on Qwen3.5 (2B, 9B and 35B-A3B preview), covering 150 languages with instruction following (glossaries, format preservation, tone). Variants add native image translation with no OCR (vision encoder), Index-Echo for speech subtitles and dubbing, Index-Homura for syllable-controlled translation and Index-NativeLong for full-document translation. The repo provides GGUF, FP8 and NVFP4 quantized builds, vLLM serving scripts, a free OpenAI-compatible API, four new benchmarks (instTrans, MEME, SandGlass, NAtIveLong), and links to three papers including HOMURA and RIVAL (RL for translation).

github.comrepoAdded 10 Oct 2026, published 15 Sep 2026🧠 AI

Halo is White Circle's open-source training framework that runs the same codebase from a single GPU to multi-node clusters, adding Expert/Context/Tensor/Expert-Tensor parallelism (EP/CP/TP/ETP) directly onto native Hugging Face models via FSDP2, DTensor and DeviceMesh rather than a separate distributed model implementation. It covers the full training lifecycle — pre-training, SFT, preference optimization, distillation, and fully asynchronous multi-turn RL with a Transformers-side trainer pushing weights over NCCL to vLLM/SGLang rollout servers — and reports up to 2.8x the throughput of stock TRL for gpt-oss-20b on 8x B300 GPUs. It ships as prebuilt Docker images for Blackwell and Hopper GPUs with FlashAttention-4, DeepEP V2, DeepGEMM, and a bf16 AdamW optimizer integrated, supports 15 MoE families, and includes LoRA/QLoRA paths that fit consumer GPUs.

github.comrepoAdded 30 Sep 2026, published 26 Aug 2026🧠 AI

DarkMoon is a self-hosted, GPLv3 AI penetration testing platform that orchestrates 50 specialist agents over an MCP-gated toolbox (Nuclei, BloodHound, sqlmap, Impacket, Naabu, httpx, ffuf, etc.) to autonomously run recon-to-exploitation across web, API, Active Directory, Kubernetes, multi-cloud, CI/CD, databases, IoT firmware, and AI/LLM inference endpoints. It uses a Privacy Gateway that tokenizes real IPs/hosts/credentials into placeholders so raw infrastructure data never reaches the (optionally local, e.g. Ollama/llama.cpp) LLM, and can generate sandbox-validated, exploit-retested pull requests for findings. It also ships a dedicated LLM agent that probes OpenAI-compatible/Ollama/vLLM/TGI inference endpoints for the OWASP LLM Top 10 (prompt injection, jailbreaks, system-prompt leakage) using garak-backed probes.

github.comrepoPublished and added 5 Aug 2026🛡️ Security

OpenHarness (Autonomous) is an open-source command center for running coding agents such as Claude Code, Codex, Cursor, OpenCode, Devin, Amp and Copilot. A native app and per-machine daemon run each agent in its own tmux pane, with a git worktree per harness. Machines are linked by password, with end-to-end encrypted traffic over direct WebRTC, Cloudflare TURN or a keyless relay. It also supports local models (Ollama, MLX-LM, vLLM, Grid) and a Store of domain-specific harnesses (Blender, CircuitJS, MuJoCo, Godogen, Strudel, RDKit, Typst) defined by a harness.json. An optional open-hardware always-on display device shows agent status.

github.comrepoAdded 16 Oct 2026, published 4 Aug 2026🧠 AI

A 10-week, 30-minutes-a-day roadmap (GitHub repo "time-to-first-token") that walks engineers through building one continuously-evolving OpenAI-compatible inference service rather than disconnected tutorials: standing up vLLM and SGLang, reading the PagedAttention/RadixAttention source, instrumenting Grafana dashboards for TTFT/inter-token latency/throughput/queue depth, load-testing past 1000 concurrent requests with GuideLLM, benchmarking FP8/INT4/AWQ quantization and speculative decoding, configuring KV cache eviction, deploying prefill/decode disaggregation on Kubernetes, and building a cost/latency/quality router. It closes with a checklist of common public-benchmark mistakes (sweep request rate, report p50/p95/p99, separate TTFT from inter-token latency, pin versions) and pointers to Stanford CS336, MIT EfficientML.ai, vLLM Office Hours, and GPU MODE for further study.

github.comrepoAdded 2 Oct 2026, published 2 Aug 2026🧠 AI

k8s-aibom is a Kubernetes controller from GoogleCloudPlatform that generates CycloneDX 1.6 ML-BOM documents at runtime for AI workloads — inference services (vLLM, TGI, Triton, Ollama), agent stacks (LangChain, AutoGen, CrewAI), RAG/vector DB infrastructure (Milvus, Qdrant, Weaviate), and training/eval jobs — by watching Kubernetes API objects with no privileged access. Each BOM attribute carries a declared/inferred/unresolved confidence flag and evidence locator, and the project positions its output as auditor-traceable evidence for EU AI Act Article 12/50, NIST AI RMF, and ISO/IEC 42001 compliance, with redaction filters, immutable GCS write-once sinks, and Sigstore-signed release attestations.

github.comrepoPublished and added 18 Jul 2026🛡️ Security

T3MP3ST is an open-source multi-agent offensive-security framework that turns an existing AI coding agent (Claude Code, Codex, Hermes, OpenCode, Oh My Pi, or a local Ollama/LM Studio/vLLM model) into an autonomous recon-to-exploit-to-report pipeline, with no separate API key required. It reports 90.1% pass@1 on XBOW's 104-challenge benchmark suite and claims 8/10 held-out post-cutoff CVEs identified down to file/line/CWE, with all headline numbers reproducible via a committed `npm run verify-claims` script. The 8-operator kill-chain design (Recon, Scanner, Exploiter, Infiltrator, Exfiltrator, Ghost, Coordinator, Analyst) maps to MITRE ATT&CK/Cyber Kill Chain phases, though only the Recon stage is described as fully live and the rest as scaffolded/experimental.

github.comrepoPublished and added 7 Jul 2026🛡️ Security

AI-Infra-Guard (A.I.G) is Tencent Zhuque Lab's open-source AI red-teaming platform that fingerprints running AI services (e.g. vLLM, LangFlow, MLflow, llama.cpp) against a library of 2000+ CVE rules, scans MCP servers and Agent Skills for vulnerabilities using the SkillTrustBench T01–T09 taxonomy, and runs multi-turn jailbreak evaluations (Many-Shot, PAIR, GOAT, ActorAttack). It also ships an LLM API poisoning/model-substitution detector, a plugin framework for community-contributed fingerprint/vuln/MCP rules, and accompanying research (FORGE-Bench, RogueHandoff-20, SkillJack) on agent loss-of-control and skill backdoors.

github.comrepoAdded 2 Jul 2026🛡️ Security

OASIS (Open Automated Security Intelligence Scanner) is an AI-powered SAST tool that runs local LLMs via Ollama or any OpenAI-compatible backend (vLLM, LiteLLM, LM Studio, llama.cpp) to detect vulnerabilities like SQLi and XSS in source code. It uses a two-phase scan (lightweight scan model + deep analysis model) orchestrated by a LangGraph pipeline (discover→scan→expand→deep→verify→report), adds a deterministic finding-validation agent that computes exploitability verdicts with confidence, and exports canonical JSON/SARIF/HTML/PDF reports with a web dashboard, RAG-backed chat assistant, --fail-on CI gating, and suppression/inline-ignore support.

github.comrepoPublished and added 28 Jun 2026🛡️ Security

Metis is an open-source agentic AI security code review framework built by Arm's Product Security Team that uses LLMs for semantic vulnerability detection rather than hardcoded rules. It performs deep security review across codebases (with C/C++ reachability analysis via a code graph), validates and triages findings from itself or third-party SAST tools to reduce false positives, and annotates SARIF output. It supports multiple LLM providers (OpenAI, Azure OpenAI, Anthropic, Gemini, AWS Bedrock, vLLM, Ollama, llama.cpp, LiteLLM) and vector-store backends (ChromaDB, PostgreSQL/pgvector, Qdrant) for RAG-based codebase indexing and querying.

github.comrepoPublished and added 31 May 2026🛡️ Security

Shannon is an open-source autonomous AI pentester from Keygraph that analyzes application source code to map trust boundaries and attack paths, then uses browser automation and CLI tools to execute real proof-of-concept exploits against the running app before reporting a finding. It only reports vulnerabilities it can actually exploit (injection, XSS, SSRF, broken auth/authz), runs in an ephemeral Docker worker with any AI provider (Anthropic, OpenAI, xAI, AWS Bedrock, Ollama, vLLM, LM Studio, etc.), and ships GitHub Action/GitLab CI integrations that produce PDF/Markdown/JSON/SARIF reports and can gate releases on exploited-severity findings. It's licensed AGPLv3 and positioned to run continuous, code-informed pentests between annual human pentest engagements.

github.comrepoPublished and added 23 May 2026🛡️ Security

Sim is an open-source (Apache 2.0) workspace for building, deploying and managing AI agents and workflows, offering a visual flow editor, 1,000+ integrations (Slack, Notion, HubSpot, Salesforce), knowledge base ingestion, and support for every major LLM plus local models via Ollama and vLLM. It ships as a cloud service (sim.ai), a desktop app, or a self-hosted Docker/Kubernetes deployment set up via an npx sim-setup wizard, and is built on Next.js, Bun, PostgreSQL/Drizzle, and ReactFlow.

github.comrepoAdded 22 Sep 2026, published 13 May 2026🧠 AI

AIMap is a Bishop Fox tool that discovers publicly exposed AI/ML endpoints (MCP servers, Ollama, vLLM/LiteLLM proxies, LangServe chains, Gradio apps, ComfyUI nodes) via 32+ curated Shodan queries, fingerprints them with Nuclei templates and live HTTP probes, and computes a 0-10 risk score based on auth, tool exposure, CORS, TLS, and system-prompt leakage. It also launches protocol-specific attack suites — MCP tool abuse, Ollama model extraction, prompt injection — with real-time streaming results and a Shodan-style query search interface plus a 3D globe visualization of findings.

github.comrepoPublished and added 6 May 2026🛡️ Security