Mapika/decider

What it claims

Decider is a family of classifiers on LLM backbones. The model reads a state and typed questions, and the answer for every question comes from one forward pass with no text generated. The page's most useful finding for someone choosing a model is that the backbone matters more than the fine-tune. On the Decision Index (v0.2.1, 70 entries, 43 benchmarks, chance-corrected), two stock instruct models read through this package's chat layout rank above the authors' trained models:

• decider-chat-gemma4-31b is #2 with 57.33.
• decider-chat-qwen3.6-27b is #8 with 51.35.
• The trained 35B, 4B and 2B all rank below both.

Similarly, the newest decider-12b started as stock Gemma-4-12B-it with no training. That stock v1 scored 0.730 on JevBench's public hard tier, equal to Jev, against 0.676 for the trained 35B.

How it works

1. decider/prompt.py renders one prompt with one answer slot per question.
2. The hidden state at each slot is projected onto one label token per option. Labels run A–J, then K–Z, then two-letter tokens, up to 255 options.
3. A softmax is taken over the valid options only, divided by a temperature from decider_config.json.

There are three ways to set that temperature:

• one global value;
• a per-type map (Choice, Noul, Score);
• since 1.8.0, temperature_by_options, which uses T(n) = max(min, a + b ln n) for a question with n options.

python -m decider.calibrate records.jsonl fits the temperatures by NLL from answers read at temperature 1.

Two prompt layouts are trained, 50/50:

• State-first is the default.
• Schema-first puts the questions before the state, so the question prefix can be cached once per schema. This is faster but less accurate, and it is opt-in.

You can also score each question in its own row. Then adding, removing or reordering questions cannot change any other answer.

Evidence

• Training cost: one epoch is 1.47M examples and 455M tokens, taking 5.3 h on a GH200 plus 45 min of evaluation.
• Reproducing v9: scripts/train.sh full matches v9 on the 94-task set (held-out 0.739 vs 0.741). The fitted temperature differs (1.03 vs 1.36).
• Gap to Jev on the Decision Index: the 35B is within 0.03 of Jev on language, retrieval, tools and arts, but 0.18 behind on knowledge (GPQA, GSM8K, CRUXEval, MMLU).
• 12B v2 vs v1: v2 adds a state-tracking LoRA (rank 32) trained on 10k rows.
• Held-out yes/no improves: 71.6 vs 58.8.
• JevBench hard tier drops: 0.712 vs 0.730.
• ECE gets worse: 0.147 vs 0.098.
• Apple Silicon (MPS, M1 Pro, fp16): median request is 133 ms vs 171 ms without the MPS patch. On held-out MASSIVE, MPS scores accuracy 0.7553 and ECE 0.0438, against 0.756 and 0.041 for the bf16 build.
• GGUF on 8 server CPU threads: a 40–120-token request takes 0.3–0.7 s on the 4B Q4_K_M and 0.12–0.31 s on the 2B. Q8_0 matches bf16; Q4_K_M is 0.2 points lower in-task.

How to use it

pip install decider-ai                  # the import name is decider

from decider.infer import Decider
d = Decider("Mapika/decider-2b")
d.decide("My card was charged twice.", [{"question": "Which team?", "options": ["billing", "technical", "sales"]}])

d.system_one(state, questions) takes typed questions:

• choice, with a criteria map;
• noul;
• score, with a list of levels.

Memory (bf16): about 4 GB for the 2B, 8.4 GB for the 4B, and 65 GB for the 35B (19.6 GB in NVFP4).

GGUF:

pip install "decider-ai[gguf]"

d = Decider("Mapika/decider-4b-GGUF", gguf_file="decider-4b-v2.1-Q4_K_M.gguf")

The GGUF files are not chat models. Loaded in llama-cli or Ollama, they behave as plain text models. The HTTP server does not serve GGUF files, and torch is still installed as a dependency.

Server: scripts/serve.sh Mapika/decider-2b 8000 exposes two endpoints:

• POST /v1/systemone, which is TypeSafe's wire format, so their SDKs work with TYPESAFE_BASE_URL;
• POST /decide, the plain form.

The server binds to 127.0.0.1. Setting DECIDER_HOST=0.0.0.0 exposes it with no authentication.

decider.serve_vllm runs on vLLM 0.29.0 and needs its own environment, because vLLM pins its own torch. It answers /v1/systemone only.

Breaking change in 1.3.0: confidence is now (n·p_max − 1)/(n − 1), not p_max. If you tuned thresholds on an earlier version, compare them against x_p_max.

Licence: Apache 2.0.

Limits

• No multi-step reasoning. There is no chain of thought, so GSM8K-style problems, temporal arithmetic and multi-hop chains are out of reach. The suggested workaround is to split the judgment into several questions.
• Hard-item calibration misses the authors' own bar. On JevBench's hard items, ECE ranges from 0.10 to 0.31 across models. On held-out generated families, v2.1 and v11 are at 0.15 and 0.16, against the 0.08 limit the authors set for release.
• Rules in the question are not followed. On a form-filling probe, a one-sentence question scores 0.67 and a paragraph of rules scores 0.24.
• Picking a JSON record by position is weak. Accuracy is 0.51 with 64 records vs 0.70 with one. Address records by key instead.
• Batched GGUF decodes are unsafe for probabilities. Packing rows with n_seq_max shifts probabilities by up to 0.16.
• English only.
• The 35B was trained with an optimizer setting since found harmful (FP32 master weights), which cost accuracy on knowledge tasks. A retrain is planned.
• Training data and code are incomplete in the package:
• The mixture-v2 builders are missing; only the public 60% of that data is reproducible.
• The RL stage (v8 to v10) is not included.
• Reproduction is not byte-identical.
• The teacher labels come from a 27B model that agreed with only 72% of its own generic-option labels.
• Known regressions exist. One example: decider-4b v2.1 drops BabyAI-GoTo from 0.54 to 0.19.
• Browser results cover only 22 click-only MiniWoB++ tasks.
• The vision variant still uses v5 weights and is being retrained.
• No leaderboard runs by the authors. Both leaderboards are third-party, and the authors only read the results.