OmniJev/awesome-jev-gallery

What it claims

The list is a curated index, current to 2026-09-29, with a companion papers-only list (Awesome JEV Papers). It is useful beyond the summary because most entries carry a headline number, a setting or a licence. Read together, those numbers give a mixed picture of Jev. TypeSafe's launch post claims 70 to 500 ms per decision at $0.042 per MTok. Its own documentation (Jev 1.13 "jaggedness") lists the failure modes: literal reading, counting, dates, indirection, distractor state and adversarial content. Independent audits put Jev below frontier LLMs on aggregate accuracy. Jev looks competitive on narrow classification and reranking tasks, at much lower cost and latency.

Evidence

All figures below are as reported by each entry's authors. The list does not re-verify them.

Aggregate accuracy against LLMs:

• The Agentpedia audit reports 67.8% for Jev against 73.1% for Opus 5.
• A secondary analysis of TypeSafe's own dashboard gives about 67.8% mean agreement against 74.1% for the best comparator.
• The dashboard itself shows 61.7–76.0% accuracy at 0.3–0.5 s per case.
• JevBench v1.3 gives Jev a composite of 74.4 on 534 decisions.

Where it holds up:

• Reranking: nDCG@10 of 0.692 against 0.691 for Cohere Rerank 4 Pro, over 8 datasets and 1,617 questions, at 422 ms.
• Jevals: on PubMedQA yes/no, 91.3% against 92.5% for the best of six LLMs, at 1/28 of the cost.
• Spam: 0.9833 accuracy zero-shot on 18,514 emails, matching a TF-IDF model trained on 14,800 labels.
• Prompt injection: 96.5% accuracy and ROC-AUC 0.9927 on 662 deepset messages, p50 325 ms.
• Moderation: 98.9% against 83.3% for the open Laya model.

Where it breaks:

• Calibration: the preregistered ChaosNLI study found that on items where annotators split, Jev's Choice confidence averages 0.807 against 0.468 human agreement. Calibration is the headline selling point, and this result cuts against it.
• Chess: no better than random at picking moves from a FEN position.
• Context compaction: 56.3% agreement, below a constant answerer.
• Judge workloads: fitted multi-dimension scoring produced 25x the hard-benign false positives of a direct question (37.2% vs 1.5%).
• Label sensitivity: a poker probe found 15–30 point swings just from relabelling the same hand.
• Agents: Jev scored 43/50 against 48/50 for a rule baseline on robot tasks. It scored 16.67% against 38.52% for the open WebJev on 125 real-website tasks.
• Benchmark hygiene: one entry (Laya vs Jev, Luni) shows that Laya's headline comparison used two different benchmarks. Check that cross-model claims share a test set.

How to use it

• Baseline first: TypeSafe's official System One Adapter serves the same typed interface from OpenAI or Anthropic models. It is the stated baseline for every comparison, so test your task through it before committing to Jev.
• Comparing models: the Jev Decision Index runs Jev and 70 open reproductions on 43 benchmarks, about 120,000 decisions per model. Scores are chance-corrected, so 0 means guessing, and a kit reruns every entrant.
• Drop-in compatibility: several open models serve Jev's /v1/systemone wire format, so they can replace the hosted API:
• Jebadiah: 27B / 9B / 4B on Qwen bases, Apache-2.0.
• WebJev: Qwen3.5-35B-A3B, Apache-2.0, trained for browser agents.
• jevos: a 619 MB GGUF that runs on CPU-only llama.cpp. It handles Noul (yes/no) questions only, at 54–220 ms and 0.815 accuracy against Jev's 0.927.
• Licences vary. Dohnuts code is Apache-2.0 but its weights are CC BY-NC-SA 4.0, which rules out commercial use. sokudan (Japanese, 314.6M) is Apache-2.0 for both code and weights.
• Training-free options: LitJev and Simple Jev read option logits from stock checkpoints after one shared prefill. They need no training, but you give up any calibration training.
• Calibration defaults: Rizzo Flow returns uncalibrated probabilities by default, even though it reports 49 ms p50 on an RTX 5060 Ti.

Limits

• Community tool security: MrJev reviewed 72 community projects and ran 66 of them in containers with a real key to see what data leaves the machine. It found a password exposed in a database URL and prompt logs readable by any user. Both were fixed upstream, but audit any tool here that holds an API key.
• Adversarial input: JevAdvBench stresses that a typed model returns a well-formed answer even under attack. Schema validity tells you nothing about correctness. The Register makes the same caveat.
• Evidence quality: many entries are small and self-reported, such as 20-row or 50-case tests. Some say outright that their timings are estimates (jevals' local runs) or rest on demos.
• Inclusion bar: an entry only has to pass one of four tests defined in CONTRIBUTING.md, which the page does not reproduce.