Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox

What it claims

Microsoft's ThinkingBox-Training uses executable checks of a task's final database state as a binary GRPO reward. It trained Qwen3.8-27B for 50 steps on 187 tasks that are disjoint from ThinkingBox-Bench. Repeatability improved: tasks passed on all 20 attempts rose from 38 (7.5%) to 107 (21.1%) of 507. Coverage fell slightly. The authors read this as "sharpening": probability moved toward trajectories that work repeatedly, and some low-probability solution paths disappeared. Their own summary is that the model became more repeatable, not broader.

How it works

Task selection runs once, offline, against the base policy. The pool stays fixed for the whole run; re-probing difficulty during training was deliberately not done.

1. GPT-5.4 screens each candidate task for solvability.
2. The base policy runs each task n times and gets k successes. Its success probability is modelled as Beta(k+½, n−k+½), a Jeffreys posterior.
3. Each task goes to the bucket holding the most posterior mass. The bucket boundaries are 0.08, 0.40, 0.60 and 0.90, set as hyperparameters. Tasks below 0.08 or at 0.90 and above are dropped, because all-fail or all-pass groups give GRPO zero advantage. For example, 3 of 8 successes gives Beta(3.5, 5.5), with 54.0% of its mass in the Difficult band.

flowchart TD
  A[Candidate tasks] --> B[GPT-5.4 solvability screen]
  B --> C[Base-policy probes, Jeffreys posterior]
  C -->|mass in 0.08-0.90| D[Fixed training pool]
  C -->|below 0.08 or 0.90+| X[Drop]
  D --> E[Group of 8 rollouts, fresh backend each]
  E --> F[Terminal-state check, reward 0/1]
  F -->|zero-variance group| E
  F --> G[GRPO update]
  G --> E

Each rollout is a full episode with its own fresh backend and isolated MCP session. GPT-5 Chat plays the simulated user. Most rewards come from deterministic state checks. A minority of narrow rubric requirements call an LLM judge, also GPT-5 Chat. Zero-variance groups are replaced in the DAPO style. Backend timeouts are refilled rather than scored as failures. Training and evaluation share one output parser (trainer/core/tb_interpret.py) so the reward measures the same agent that evaluation measures.

Training setup:

• full-parameter, token-mean GRPO
• 18 groups × 8 rollouts (144 episodes) per step
• 24 H100s, synchronous
• parameter and optimizer offload
• AdamW, learning rate 1e-6, weight decay 0

Evidence

All numbers come from 507 benchmark tasks × 20 attempts. The benchmark's domains, databases and most tools were unseen in training.

• Fine-tuned versus other models: GPT-5.6 Sol scores higher on pass@1 (61.91%) but passes fewer tasks 20/20 (82). Claude Sonnet 4.6 scores 59.19% pass@1 with 102 tasks at 20/20. Claude Opus 5.5 leads with 241 tasks at 20/20.
• By domain: pass@1 and 20/20 rose in all five domains. pass@20 fell in retail, travel and auto insurance; auto insurance fell most, by 7.0 points. Retail had the largest 20/20 gain, +24.5 points.
• Cost per dependable task: $8.00 for the fine-tuned model, against $6.80 for GPT-5.4, $7.45 for GPT-6 Astra and $7.80 for Claude Opus 5.5. These figures are a comparative serving index and exclude training and probing costs.

How to use it

Requirements: Linux or WSL, Python 3.12, uv, Verl 0.9.0 (patched), a local Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for the simulated user and judge. The reference topology is 3 nodes × 8 H100.

git clone https://github.com/microsoft/thinkingbox.git
git clone https://github.com/microsoft/thinkingbox-data.git
git clone --branch main https://github.com/microsoft/thinkingbox-training
git -C thinkingbox checkout 892964e7226044e5463ad188da119df838f4bec1
git -C thinkingbox-data checkout thinkingbox-bench-v1.0

python scripts/prepare_verl.py --dest .deps/verl --install
python scripts/verify_verl_install.py
uv pip check

Do not run another automatic package sync after these steps: it can replace the patched Verl with the vanilla wheel. python data/prepare_data.py runs probe aggregation, difficulty filtering and benchmark exclusion. A dry run must exit zero before you allocate GPUs:

DRY_RUN=true ./scripts/run_train.sh \
  trainer.total_training_steps=50 \
  trainer.save_freq=5

Evaluation merges the checkpoint and serves it with vLLM; the historical run used 8 A100s at tensor-parallel 8. It then runs thinkingbox-eval --repeat 20 --coverage-profile full through OpenEnv. The code is MIT-licensed, and the benchmark data is CDLA-Permissive-2.0.

Limits

• The 187-task training pool is private. The public sample list only exercises the interface, so you must supply your own task source, disjoint from the benchmark.
• The transfer claim is bounded. Training and benchmark tasks share the framework, the simulated-user setup, the grading method, some tools and a similar task style.
• Coverage (pass@20) declined overall.
• Only one model and one 50-step run are reported.
• Configurations smaller than the reference setup are not validated.