y0usaf/pi-jev

What it claims

pi-jev is a Pi extension that routes pending tool calls and their outputs through a separate hosted judge (TypeSafe Jev at api.typesafe.ai) that answers typed questions with numbers, not prose. Beyond the design the summary names, the page establishes the operational contract: all four judgements ride in one request (~300 ms round trip), the default is shadow mode (flag and notify only), every error path fails open (missing key, timeout, 429, or malformed response → no verdict, call proceeds), and errors are throttled to once a minute. It ships with no runtime dependencies; Pi imports sit in peerDependencies, and src/client.ts runs standalone under node --input-type=module.

How to use it

pi install npm:@y0usaf/pi-jev

Config lives at ~/.pi/agent/pi-jev.json or project-scoped .pi/pi-jev.json (project keys win, per-key override). Key resolution order: TYPESAFE_API_KEY env, apiKey in config, then apiKeyFile path. Defaults include mode: shadow, gate tools: ["bash","write","edit"], cacheSeconds: 120, and blockOn values destructive 0.9, exfiltration 0.7, beyondScope 0.85, impact 2.5 (impact is a 0–3 damage rubric). Slash commands: /jev (status), /jev on|off, /jev mode shadow|enforce, /jev last, /jev output, /jev check <text>.

The data boundary

Worth knowing before adopting: each judgement sends the working directory, tool name, the last user message (first 1200 chars), and the tool arguments to api.typesafe.ai — and for write/edit, those arguments contain file content. Strings over argumentChars (400) are truncated with a marker and the elided text stays local. To keep file bodies out entirely, set gate.tools to ["bash"]. The API key travels in the Authorization header and is scrubbed from transcript notifications.

flowchart TD
  A[pending tool call] --> B[gate: 4 questions, 1 request]
  B --> C{flagged?}
  C -->|no| R[run]
  C -->|shadow| N[notify + footer, run]
  C -->|enforce| P[confirm before run]
  B -.error/no key/429.-> R
  A -. write/edit args = file content .-> X[(api.typesafe.ai)]
  B --> X

Evidence

The output judge was calibrated over 53 fixtures × 3 runs (203 requests, 0 failures, 490 tokens/request, 126 ms median). The leak question showed no overlap — ≥0.92 for leaks against ≤0.02 otherwise — so the 0.90 threshold sits in an empty band. The gate thresholds come from a smaller "smoke calibration" (six states, a few runs each), which the author explicitly flags as not a labelled eval. Three findings shaped the design: an ordinary requested edit scores up to 0.85 on destructive (hence the 0.90 threshold, not 0.70); beyond_scope separates in-scope 0.10–0.74 from out-of-scope 0.90–0.98 with the threshold mid-gap at 0.85 because runs jitter ±0.05; and asking whether an action "cannot be recovered from version control" scored a real rm -rf src && git push --force at only 0.77, while asking plainly whether it is destructive separated the pair 0.03 vs 0.99.

Limits

The gate sees only intent — it cannot catch a secret echoed to the transcript or distinguish a network hiccup from a type error, which is why the output judge exists. Two retry-shaped questions were tried and dropped for overlapping phrasings (e.g. git commit --amend --no-edit scored 0.68 on "safe to run again"), so failure advice is derived from a class table in code rather than asked. Headless runs (-p, RPC) cannot prompt, so enforce mode falls back to a warning unless gate.blockWithoutUI is set. The author states the gate calibration is deliberately not strong enough to switch enforce on by default.