Fast Models, Slow Evidence: Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses

What it claims

On 11 typed harness decisions, the hosted System-1 model JEV (jev-1.13.0) beats open-weight LAYA (0.3.21) overall: 76.7% vs 63.1% on the 2,100-case suite (A–F) and 80.2% vs 66.0% on the 1,280-case agent-infrastructure suite (G–J). The paper's more useful claim is that these models save much less as gates than naive accounting suggests. A JEV safety pre-screen cuts review cost by only 4.3%, and its real benefit is latency. RAG skip-gates are capped by the share of irrelevant retrievals in traffic. Thresholds tuned in-sample overshoot their miss targets on held-out data.

How it works

Both APIs take a state string and a dictionary of questions. A choice question has instructions plus a criteria map from option keys to short descriptions. A yes/no question (noul) returns p(yes). LAYA routes each request to an English or a multilingual checkpoint.

Confidence is computed as max p for choice questions and max(p, 1−p) for yes/no. LAYA's own confidence field is an entropy-based quantity on a different scale. It differs from max p by 0.299 on average (JEV: 0.010), so thresholding on it silently breaks.

Statistics use exact McNemar tests on paired cases and Wilson intervals. DeepSeek-V4.1-Flash at temperature 0 serves as a reference LLM in the control experiments only.

LAYA's tool-selection collapse has a mechanical cause in its released code. All options share a 192-token budget, and each option is truncated to max(4, (192−16)/K) tokens, so at K=50 an option is about 4 tokens long.

Evidence

Determinism

• LAYA changed 0 of 420 decisions on re-run and reproduced 1,180/1,180 decisions across an RTX 2080 Ti and an Apple M4 CPU (max |Δp| 0.006).
• JEV changed 1 of 420 on re-run and 5 of 1,080 a day later.

Per-track accuracy and error direction

• Large label sets (F): JEV 81.0% vs LAYA 35.0%. LAYA scored 0/10 in Arabic and Hindi with 60 labels.
• Model routing (A): both at chance. JEV answers "cheap" 99.5% of the time; LAYA's AUROC is 0.48.
• RAG relevance (D, n=1,920): a tie, 61.0% vs 60.5%, but the two err in opposite directions. JEV recovers more relevant passages (58.3% vs 45.4%); LAYA rejects more BM25 hard negatives (61.8% vs 49.3%).
• Tool-call guard: LAYA calls a tool on 74.7% of requests that need none (JEV: 40.0%).
• Pre-execution safety: LAYA passes 81.5% of harmful requests (JEV: 25.5%).
• PII egress: LAYA's AUROC is 0.41, so its scores rank the classes inverted.

Tool selection vs number of candidates

• Random distractors at K=50: LAYA 53.8%, JEV ≥98.8%.
• Nearest-neighbour distractors at K=50: LAYA 31.3%.
• On items with a unique correct tool: JEV 98.9% vs LAYA 61.6%. In 19.8% of these items an LLM audit found a second valid tool.

Robustness and calibration

• Reversing option order changes 30.3% of LAYA's answers (53% with 60–77 labels) vs 1.8% for JEV.
• Rewording the instructions moves both models 5–15 pp per track, so every accuracy figure carries roughly a ±5 pp wording band.
• 26% of LAYA answers with confidence ≥0.9 are wrong.
• Platt scaling on half of track D drops LAYA's ECE from 0.22 to 0.027 on the other half.

Injection controls

• The apparent "channel effect" on false positives vanishes with text that naturally appears in tool outputs.
• With payloads embedded in documents, LAYA detects 47.5–50.8% (70.8–82.5% standalone). JEV detects 90.8–93.3% embedded and 88.3–94.2% standalone.

Gating

• RAG skip-gate at a 5% miss budget: LAYA keeps 92.4% of LLM calls, JEV 79.8%. A perfect gate saves at most 58% in this mix.
• On real top-5 contexts, both models recognise fewer than 40% of "no answer here" cases.
• Thresholds picked on one half and tested on the other: the 95th-percentile miss rate is 6–8% on D (800 positives) and 10–17% on tracks with 100–200 positives, against a 5% target.
• Two-tier safety review with JEV as pre-screen:
• At τ=0.1, 65.6% of traffic still reaches the LLM reviewer and cost falls only 4.3%. At τ=0.05 the system costs 4.7% more than the reviewer alone.
• With held-out thresholds, the pre-screen adds at least one extra miss in 47% of splits.
• LAYA must pass 99.3% of traffic to keep recall.
• LAYA→JEV cascade (escalate below 0.9 confidence): keeps 41.7% of decisions local and saves $16 per million decisions, but accuracy drops from 76.7% to 70.7%.

How to use it

Cases, raw outputs and analysis code are at https://github.com/David-DL-Space/sys1-eval.

Latency depends on deployment:

• LAYA: 31 ms p50 on an RTX 2080 Ti, 200 ms on an M4 CPU.
• JEV: 325 ms or 83 ms p50 depending on client site.
• The first version's "10× faster" claim reflected one network path.

Cost:

• JEV uses 759 tokens per decision, about $32 per million decisions at $0.042/M tokens.
• The cheap batched LLM reviewer costs $72 per million requests, so one JEV call is 20% of a review.

Recommendations stated by the paper:

• Use a RAG gate to trigger more retrieval, not to skip generation.
• Calibrate thresholds on several hundred positives, or add a conservative margin.
• Count the pre-screen's own tokens in any cost estimate.

Limits

• No multiplicity correction is applied. Every claimed difference has p<0.001.
• The RAG traffic mix is synthetic, so absolute savings will differ in production; the ranking (JEV AUROC 0.71 vs 0.61) is expected to hold.
• Safety gold labels come from GLM-4.6 applying a written policy and agree with the dataset labels on only 72.8% of 540 items.
• Routing labels derive from RouterBench outcomes of a 7B model vs GPT-4.
• The fetched text stops partway through the audit table (Table 2) and does not include the label-noise section (§8).
• The page states no licence for LAYA's weights.