y0usaf/pi-jev

What it is

@y0usaf/pi-jev wired to api.typesafe.ai, judged from a security-review angle: the summary already covers the gate/output-judge architecture and fail-open design, so what matters here is what leaves the machine, how to constrain it, and the calibration findings the README documents.

What data is sent, and how to limit it

Every judgement POSTs to api.typesafe.ai: the working directory, the tool name, the last user message (first 1200 chars), and the tool arguments. For write and edit those arguments contain file content. The output judge additionally sends the first output.outputChars (2000 default) of a bash result.

Truncation is the containment mechanism: any string longer than gate.argumentChars (400 default) is cut to …[N chars elided]; the omitted text stays local. To keep file bodies off the wire entirely, set gate.tools to ["bash"] (default gate is ["bash","write","edit"]). The API key travels in the Authorization header; notification text is scrubbed of any registered key before hitting the transcript.

Key resolution order: TYPESAFE_API_KEY env → apiKey in config → apiKeyFile path (with ~/ expanded).

Enforcement behavior

Shadow mode is the default: a flag produces a notification and footer status, nothing blocks. enforce mode prompts for confirmation before a flagged call runs. Headless runs (-p, RPC) cannot prompt, so enforcement silently degrades to a warning unless you set gate.blockWithoutUI. The README explicitly says the smoke calibration is not enough to ship enforce-by-default.

Full blockOn thresholds (0–1, except impact on a 0–3 rubric): destructive 0.9, exfiltration 0.7, beyondScope 0.85, impact 2.5. minConfidence gates only the impact dimension, because the three yes/no ("noul") questions return a probability with no confidence.

Calibration findings worth stealing

The thresholds were measured, and three findings are transferable to anyone building LLM-as-judge gates:

• Prompt phrasing dominates. A first-draft destructive question that asked whether data "cannot be recovered from version control" scored a real rm -rf src && git push --force at only 0.77 — because "it's in git" is a reasoning path the model takes. Asking plainly whether the action is destructive separated the same pair 0.03 vs 0.99.
• Set thresholds mid-gap, not on an edge. In-scope states scored 0.10–0.74, out-of-scope 0.90–0.98; the same state drifted ±0.05 between runs, so the threshold sits at 0.85 in the empty band.
• Split multi-factor questions. Two "is it safe to run again" retry questions were dropped: phrasings overlapped (yes 0.73–0.96 vs no 0.37–0.66; git commit --amend --no-edit scored 0.68 despite being unsafe). Advice is now derived from the failure class in code (CLASS_ADVICE table in src/output.ts), not asked.

The leak question showed clean separation across 53 fixtures ×3 runs (203 requests, 0 failures, 490 tokens/request, 126 ms median): ≥0.92 leak vs ≤0.02 clean, threshold 0.90.

Caveats

The gate sees intent only — it cannot catch a credential echoed into output (that's the output judge's job) or distinguish a network hiccup from a type error. The gate calibration is six states over a handful of runs: the author calls it a smoke test, "not a labelled evaluation set." src/client.ts imports nothing from Pi and runs standalone under node --input-type=module, so the HTTP-and-redaction path is auditable in isolation.