vinilana/jev-gateway

What it adds beyond the summary

The summary names the routing idea; the page gives the install path, the per-client mechanics, and the actual benchmark numbers — plus a fail-open design and one exception to it.

How to use it

npm install -g jev-gateway

Then launch your agent through a wrapper (jev-codex, jev-claude, jev-opencode, jev-kilo, jev-gemini, jev-devin), used exactly like the underlying CLI — every arg passes through. Needs Node.js 22.15+. First run asks once where to reach Jev and for a key, checks it with one real call, and saves to ~/.jev-gateway/.env. Watch routing with jev-codex --dashboard (Codex 8790, Claude 8789, OpenCode 8791, Gemini 8788, Devin 8792, Kilo 8793; override per-client with JEV_*_PORT). Toggle --routing off/on to A/B the same work. MIT licensed. Standalone server mode (pnpm dev, port 8787) exposes POST /router/decide to ask Jev for a decision directly, and x-jev-gateway: off skips Jev per request.

Jev is reachable through four billing paths that return identical answers: TypeSafe (official), OpenRouter, Vercel AI Gateway, OpenCode (defaults to free jev-1.13-free; paid is jev-1.13).

How it works

Each tool-carrying turn triggers one Jev call; the conversation is the "state" and Jev answers typed questions (which tool or none, whether a tool is needed at all as a cross-check, and closed-set argument values), returning calibrated probabilities. That picks a mode reported in x-jev-gateway-mode. Tool lists over 120 entries (Claude Code sends ~280) take two Jev calls: rank in groups, then decide among the top 3 per group. Codex agent_message items pass through unrouted, and once one appears it persists for the rest of the conversation — so subagent sessions are never routed.

Client mechanics differ sharply, and this is the part the summary flattens:

• Codex speaks the Responses API; forced tools work fully.
• Claude Code is hint-only — the Anthropic API rejects a forced tool while extended thinking is on, and changing tool_choice would invalidate its per-turn cache. "Expect better tool picks on large tool lists, not lower cost or latency."
• OpenCode/Kilo route only jev-gateway/* models; agents naming their own model bypass the gateway, and the launcher lists which ones before start. Forced mode only helps if the chosen model actually honors tool_choice — free Kilo models tested did not reliably obey.
• Devin uses Connect RPC/protobuf, decoded schema-less; steering is always hint or a synthesized stream.

flowchart TD
    A[Agent turn with tools] --> B{agent_message present?}
    B -->|yes| P[Passthrough, rest of convo]
    B -->|no| C[Ask Jev]
    C --> D{Confident + client supports force?}
    D -->|yes| F[Force/steer tool]
    D -->|hint-only client<br/>Claude, Devin| H[Append suggestion]
    D -->|low confidence / Jev down| L[Untouched to LLM]

Evidence

jev-gateway-bench: 120 sessions, six models, two chess tasks (debug + feature), five runs/mode. Routing helped debugging for every model. On the feature task it made Opus 5 and Sonnet 5 clearly worse, and GPT-5.6 Luna got cheaper but solved 3/5 runs vs 5/5 without routing.

A separate 20-run OpenCode series (Opus 5, forced tools via Chat Completions): median output tokens −62% and time −79% on bugfix; −29% and −31% on feature; all 20 passed hidden checks. But gateway version, provider, tool roster and date all differed from the Claude runs, so it doesn't isolate the harness.

Limits

• Fail-open by design: if Jev is down/slow or the key is wrong, requests go straight to the LLM; a rejected rewrite (400/422) resends the original.
• The one exception: Devin's Connect streams report errors after HTTP 200, so a refused hint reaches Devin as a failed turn — set JEV_ROUTING=off if it happens.
• A wrong forced tool can derail a turn (raise JEV_MIN_CONFIDENCE); Jev adds ~0.5–1s per tool-carrying turn.
• Jev reads text only, 32k window, most accurate in English; images become placeholders.
• Five runs per cell is a small sample (author's own caveat) — measure on your own work.
• OpenRouter, Vercel, and the Gemini path have tests but have not been run against real keys/APIs. OpenCode v2 and Gemini Google-account login are out of scope. Not affiliated with TypeSafe.