Think Before You Act: Decision Gate for LLM Agents

What it claims

JevAct, from IBM SIL (arXiv v1, 28 October 2026), puts a gate in front of an unmodified ReAct agent. The gate checks write calls and hand-offs before they execute. Each check combines generated Python over tool records with yes/no questions answered by Kev-27B, an open-source decision model that scores fixed answer options in one forward pass. The largest gain is on τ²-bench Telecom, where Claude Opus 4.6 goes from 39.5 to 80.7 pass^3.

How it works

flowchart TD
  P[Agent proposes message] --> T{Write call or hand-off?}
  T -- no --> C[Commit as in ReAct]
  T -- yes --> K[Code checks + Kev questions + Boolean rule]
  K -- pass --> C
  K -- flag, budget left --> F[Discard; one-shot feedback]
  F --> P
  K -- budget spent --> C

Offline, an LLM compiler turns the policy and tool schemas into checks. Each write tool gets yes/no questions, a Boolean rule over the answers, and a Python function that computes facts such as dates, amounts and counts from tool results. Hand-offs get a checklist of problem types and the steps required for each. The key compile-prompt rule is "a condition that was never discussed is not ruled out." The compiler gets one repair pass on unlabeled development logs. Questions that Kev confidently flags on more than 10% of a tool's development calls are dropped. Checklist steps flagged on more than 5% of hand-offs in solved training episodes are also dropped.

At runtime, a write gate answer counts as "no" only when P(yes) ≤ 0.05. A checklist step is open when P(required) ≥ 0.5 and P(settled) ≤ 0.5. Revision budgets are R=2 for writes and R=1 for hand-offs. A flagged proposal is never executed or stored, and the feedback is not added to the history. BFCL has no policy, so it uses two generic checks. The first is an end-of-turn check: when a tool error is still unresolved, it suggests a tool if P ≥ 0.6. The second is a requested-action gate with a 0.05 threshold.

Evidence

Each pair below is pass^3 (success in all three runs), ReAct then JevAct, with the same agent model in both arms:

On Telecom, all eight agents tested improve on both at-least-once success and pass^3. On BFCL, all nine improve on pass^3, but at-least-once results are mixed. Airline effects are smaller and vary by agent, so the four rows above overstate the Airline result.

The test settings are Airline with 50 tasks, Telecom with its 114-task base split, and BFCL multi-turn with 800 entries. The τ² user simulator is GPT-4.1. Agents run at temperature 0 with at most 4,096 output tokens.

How to use it

The page lists these inputs: a written policy, tool schemas, and logged tool calls and outputs from development runs, which need no outcome labels. Kev-27B is a fine-tune of Qwen3.8-27B and is used without further training. Any model that answers the same fixed-choice questions can replace it. The agent's LLM, prompt and tools stay unchanged. A check adds at most R extra agent calls, and judge cost does not grow with the number of questions. The fetched text gives no repository, install command or licence.

Limits

The comparisons test the whole harness (compiled checks plus revision chances), not the judge in isolation. The authors state that Kev's probabilities are not guaranteed to be calibrated. Once the budget is spent, the next proposal is committed without a check, so the gate cannot block an action it has already flagged twice. The fetched text ends partway through the setup section, so it does not include the component and judge ablations or the judge-cost estimates the paper reports. Eleven agents are tested overall, but only six run on all three benchmark settings.