AnotiaWang/awesome-decision-models

What it claims

The list is the main index of the "decision model" ecosystem around TypeSafe AI's Jev, which launched in September 2026. Its value over the summary is in the later sections, which the summary did not see. They cover the Benchmarks and research and independent evaluations sections, cookbooks, and many concrete limits, licences and install commands. The independent evidence it collects is consistent:

• Jev's clearest advantage is latency and cost, not accuracy.
• Its probabilities are useful for gating but not reliably calibrated.
• Its input should be treated as untrusted.

Evidence

All figures below are as reported by the linked projects.

Accuracy against LLMs

• Banking77, all 3,080 test utterances (OpenRouter): Jev 1.13 vs Claude Opus 5.
• Accuracy: 81.0% vs 84.4%.
• Median latency: 175 ms vs 2,266 ms.
• Cost: about $0.11 vs $2.42 per 1,000.
• 791 labelled decisions: Jev matches small models and trails GPT-5.6 Terra by about 5 points on 77-way routing. A 0.80 confidence gate that escalates the rest to Terra matches Terra's accuracy at about a quarter of the cost.
• LexGLUE, 7 tasks, 23,607 examples, zero-shot: mean micro-F1 69.9 at $4.02, against 71.3 at $16.45 for GPT-5.6 Luna via chat JSON.
• 37 datasets, 346,009 requests, under $10: Jev 1.13 beats Qwen3.8-27B option probabilities on 27 of 37 and Gemma-4-E4B on all 37. Choice probabilities are well calibrated; binary thresholds are poorly placed.

Calibration and probability coherence

• Probability axioms: P("X") + P("not X") misses 1 by 0.064 on average, against 0.293 for Qwen3.8-27B's first-token readout. Single-label probabilities over-sum to 1.14.
• Fair dice: Jev picks face 1 on all 400 rolls, with 82.9% mean reported probability against 19.0% accuracy.
• Evidence audit of 28 early papers: the typed readout shows no independent accuracy advantage over comparable label-probability readouts. The paper gives a 14-item evaluation checklist.

Where Jev loses or breaks

• Adversarial (JevAdvBench, 812 questions, 9,744 single-edit attacks): appending one unverified opinion to the state flips 12.1% of Jev 1.13 decisions.
• Phishing (2,000 emails): Claude Haiku 4.5 beats Jev on accuracy.
• Search rerank (9,831 pairs): Jev alone does not beat embeddings (BM25 / bge-m3); fusion wins.
• Agent security: good aggregate calibration hides confident failures concentrated in particular attack groups.

Open models (self-reported)

• pplx-decider-v1-27b: 85.71% mean over 11 benchmarks vs 84.51% for Jev.
• JEV-27B: 84.07 vs 83.85 over six benchmarks.
• WebJev: 38.5% vs 16.7% task completion for Jev 1.13 on 125 real-website tasks.
• jevos: about 54 ms per short request on CPU.

The list warns that these open-model scores are not comparable across entries.

How to use it

The common production pattern across many entries (jev-harness, the 791-decision study, JevLoop, toolgate, jev-secret-guard) is a confidence gate with deterministic code owning thresholds:

flowchart LR
  S[State + typed questions] --> J[Decision model]
  J --> G{p >= threshold?}
  G -- yes --> C[Code acts]
  G -- no --> L[Escalate to LLM or human]

Install commands given on the page:

pip install typesafe-sdk
npm install @typesafe-ai/sdk
pip install system-one-adapter
pip install "jev-style[torch]"

• Avoid the typesafe-ai PyPI package. It is a defensive redirect shim; install typesafe-sdk instead.
• Ollama serves /v1/systemone from version 0.35, with nimble and tev1 in its library.
• jevcal fits per-question thresholds to a target accuracy on your labelled data, and fails CI when a Jev update breaks them.
• wellposed lints requests offline with 40 structural checks.

Limits

Hard request limits

• meraGPT Decider 1: 4,096 tokens per request and ten Choice labels.
• Jev: a 255-option Choice cap (jev-tree works around it).
• jevos: Noul only; Choice and Score return 422.

Licences

• Apache-2.0: Clef, pplx-decider, JEV-27B, WebJev, Winnow, JevK5, Standard One.
• Non-commercial: OpenJev weights (CC BY-NC 4.0) and Vev weights.
• The list itself is CC0.

Access and maturity

• OpenAI Decisions API is a limited preview with unpublished pricing.
• Solar Decide is in beta.
• OpenRouter's Decisions API is alpha and cannot be called from chat-completions SDKs.

Comparability

• Jev Decision Index notes that open-model inference timings and Jev's hosted HTTPS latency are not directly comparable.

Independence

• Most studies measure only Jev, and many project numbers are author-reported on chosen benchmarks.