eliasbiondo/antirion

What it claims

Antirion is an AI gateway written in Rust. The gateway, the Admin API and the web console ship as one binary or one distroless image, licensed Apache 2.0 (copyright 2026 Elias Biondo). The author claims sub-millisecond added overhead at thousands of requests per second on 4 vCPUs. The author also claims several times the throughput of the next fastest gateway on the same machine, and that it keeps serving at rates where LiteLLM fails.

How it works

Each request on the data plane (port 8080) runs through a fixed pipeline: authenticate (virtual keys, JWT, bring-your-own provider keys), admit against rate limits and budgets, input guardrails, exact or semantic cache lookup, routing, the attempt (retries, hedging, circuit breakers, dialect translation), then output guardrails and the cost ledger. The record is written once the request settles. The admin plane (port 8081) serves the console and Admin API. Configuration changes are committed as numbered generations, with diffs and rollback, and applied to every node without a restart.

Routes are trees of nodes: fallback, weighted (smooth weighted round robin, random, or Thompson sampling), least_cost, least_latency, least_busy, conditional, classifier and cost_quality. Model tiers (frontier, balanced, fast, specialized) sit alongside these nodes. POST /routes/{name}:simulate shows which deployment a request would reach and why. Streams are translated event by event, so an OpenAI Chat client can call Claude and an Anthropic Messages client can call GPT. Plugins hook each request stage as signed webhooks or WebAssembly components.

State lives in an embedded store on a standalone node. A cluster shares PostgreSQL for state and Valkey for counters, budgets, cache and health.

Evidence

Antirion, Bifrost 2.2.4 and LiteLLM 1.103.1 ran interleaved on dedicated AWS c7i machines. Each gateway checked one key per request, with no request logging and no cache. The headline scenario is simple chat at 5,000 requests per second, with the gateway on one c7i.xlarge (4 vCPUs). The load generator and the upstream each ran on their own c7i.4xlarge in the same placement group. Load was open loop, so queueing shows in latency, and the upstream answered at once. Figures are medians of repetitions. The latency, CPU and memory numbers are in charts that are not in the page text. The harness is in bench/ and the method is in docs/benchmarks.md.

How to use it

Zero-config start; the first run prints the console URL and a one-time admin token:

docker run -p 8080:8080 -p 8081:8081 -v antirion-data:/var/lib/antirion eliasbiondo/antirion

OpenAI SDKs point their base URL at http://localhost:8080/v1; Anthropic SDKs at http://localhost:8080. A lost token is recoverable from the volume:

docker cp <container>:/var/lib/antirion/data/bootstrap-token - | tar -xO

Production runs should set ANTIRION_BOOTSTRAP_TOKEN and ANTIRION_ENCRYPTION_KEY, because the key seals stored provider keys. Set ANTIRION_PUBLIC_ORIGIN when the console is served at another address. Compose files cover one node (docker compose up -d) and a two-node cluster (compose.cluster.yaml), which needs a bootstrap token of at least 32 characters. A Helm chart is at deploy/helm/antirion; --set mode=cluster alone bundles PostgreSQL and Valkey for trials. The binary also offers antirion validate -f <file>, antirion schema and antirion healthcheck.

Limits

Release binaries are Linux only (glibc 2.36 or later, amd64 and arm64). macOS and Windows users must use the container or build from source, which needs Node 22+ and pnpm for the console. The benchmark used a minimal configuration with no logging, no cache and no guardrails, so overhead with those features on is not shown. All performance claims come from the project's own harness.