Bifrost (Maxim AI) — open-source LLM gateway

What it claims

Bifrost is a Go LLM gateway that you can run as a standalone HTTP server or embed as a Go library. It is licensed under Apache 2.0. Requests use a provider/model naming scheme, for example openai/gpt-4o-mini. The page reports 11 µs of added overhead per request in a sustained 5,000 RPS benchmark, with a 100% success rate.

The open-source repo is not the whole product. The page says "enterprise deployments unlock" these features:

• adaptive load balancing
• clustering
• guardrails
• MCP gateway
• private networking and custom security controls

Check which features you need before assuming they are in the Apache-licensed build.

How it works

The repo has four layers:

• core/: provider implementations and shared schemas.
• framework/: pluggable storage backends for config (configstore), request logs (logstore) and vectors (vectorstore). The vector store backs semantic caching.
• transports/bifrost-http: the HTTP gateway, plus a web UI in ui/.
• plugins/: middleware modules:
• governance (budgets, access control)
• semanticcache
• logging
• telemetry
• jsonparser
• mocker (mock responses for testing)
• maxim (integration with the vendor's own observability product)

flowchart LR
  A[Existing SDK via /openai, /anthropic, /genai] --> T[bifrost-http transport]
  B[Native /v1/chat/completions] --> T
  T --> C[core]
  G[Go app: go get .../core] --> C
  C --> P[plugins: governance, cache, logging]
  P --> R{provider + fallback}
  R --> O[OpenAI]
  R --> N[Anthropic / Bedrock / Vertex / ...]

Evidence

Performance figures from the 5,000 RPS sustained benchmark:

• 11 µs mean added overhead per request, described as "less than 15 µs"
• sub-microsecond average queue wait
• about 10 ns to select a weighted API key

The README does not give the hardware, payload, or whether upstream providers were real or mocked. Those are in a separate "Performance Analysis" doc that is not included here.

How to use it

Run the gateway, then configure providers in the web UI at http://localhost:8080:

npx -y @maximhq/bifrost
docker run -p 8080:8080 -v $(pwd)/data:/app/data maximhq/bifrost

Mount the /app/data volume if you want configuration and data to persist across restarts.

To embed Bifrost in a Go program instead of running the gateway:

go get github.com/maximhq/bifrost/core

To migrate existing SDK code, change the base URL to a provider-specific path prefix:

# OpenAI SDK
base_url = "http://localhost:8080/openai"
# Anthropic SDK
base_url = "http://localhost:8080/anthropic"
# Google GenAI SDK
api_endpoint = "http://localhost:8080/genai"

The page also lists integration guides for the AWS Bedrock SDK, LiteLLM and LangChain. Full documentation is at https://docs.getbifrost.ai.

To call the native OpenAI-style endpoint directly:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Hello, Bifrost!"}]}'

You can configure the gateway through the web UI, through its API, or from a file.

Limits

• Feature split: The README lists clustering as a doc topic but also names it as enterprise-only. Confirm what multi-node deployment the OSS build supports before you plan HA around it. The same check applies to MCP:
• The feature list includes MCP tool use.
• The enterprise list includes the "MCP gateway".
• Benchmark scope: The performance numbers come from a single 5k RPS setting, and the README does not describe the methodology.
• Vendor plugin: One bundled plugin (maxim) ties into the vendor's commercial observability product.