LiteLLM

What it claims

LiteLLM is a self-hosted gateway that sits between an organization's apps and its model providers. Every request passes through it, so it can enforce access, attribute cost, stop over-budget traffic, and pick a deployment. The page's headline new item is a Rust implementation of the gateway. The vendor says it adds 0.66 ms at p99, 3.5× lower overhead than the next AI gateway. The page also claims 240M+ Docker pulls and quotes one customer saying coding-task costs fell "by as much as 56%". That quote is unattributed and has no setting, so treat it as marketing.

How it works

The page describes this order of controls on each request:

flowchart LR
  A[App / agent] --> K[Virtual key / SSO / JWT]
  K --> B{Budget or rate cap hit?}
  B -- yes --> X[Request stopped]
  B -- no --> C{Cache hit?}
  C -- yes --> R[Cached response]
  C -- no --> RT[Router: load balance / lowest-cost / auto]
  RT --> P[Provider, MCP, agent, or self-hosted model]

• Budgets are hard caps per key, team, org and model, with daily or monthly resets. At the cap, requests stop rather than just raising an alert.
• Auto Routing sends simple prompts to cheaper models and hard ones to stronger models. The page does not say how it judges difficulty.
• Prompt compression is listed but not explained.
• State lives in your own Postgres and Redis. The gateway scales out with Kubernetes autoscaling.

Evidence

The benchmark setup, as stated:

• every gateway ran on identical hardware;
• all were pointed at the same deterministic mock upstream;
• a single client sent the traffic;
• resident memory was measured with the gateway idle in the same run.

The tool is AI Gateway Bench, which the page calls an open standard, and it says all scripts and charts are public. The page does not name the comparison gateways, the request rate, or the memory figures. A single client with a mock upstream measures pure added latency, not behavior under concurrent load.

How to use it

• Deployment options: official Helm chart, Terraform module, Docker images (including database-bundled and non-root variants), one-click deploy on AWS, GCP or Azure, and fully air-gapped installs. The page gives no commands.
• Logging integrations: Langfuse, Arize Phoenix, LangSmith, OTEL.
• Secrets: read from AWS Secrets Manager, Vault, or Azure Key Vault.
• Guardrails: Presidio and Lakera for PII masking and prompt-injection checks.
• Security-review items:
• Cosign-signed, hardened non-root images;
• Grype scans (vendor claims zero high or critical findings) and CodeQL on the code;
• SCIM provisioning;
• no telemetry when self-hosted.
• Releases: there is a stable release track whose images ship after load testing. The changelog and security disclosures are public.

Limits

• Open-source vs. enterprise split. This matters before adopting, and the summary blurs it.
• The MIT open-source tier includes: provider integrations, virtual keys, budgets, teams, load balancing, RPM/TPM limits, guardrails, and logging integrations.
• SSO, JWT auth, RBAC and audit logs are listed under the paid enterprise tier, along with support and SLAs.
• If your rollout depends on SSO or audit logs, budget for a licence.
• Benchmark source. The 0.66 ms figure is the vendor's own run. The page invites you to rerun it, and you should before relying on the number.
• Rust gateway coverage. The page does not say whether the Rust gateway supports the full feature set listed elsewhere on the page.