Teaching LLMs to Update Beliefs for Efficient Long-Horizon Interaction

What it claims

ABBEL trains an LLM agent to act from a periodically rewritten natural-language belief state instead of its full history. The new part is belief grading: an auxiliary RL reward that supervises what each belief contains. With a generic reconstruction grader on CollabBench, ABBEL closes about 50% of the gap to full-context models and trains in about 50% fewer steps than summarization without grading. On Combination Lock, a grader built from domain knowledge lets ABBEL match or beat full context in learning efficiency.

How it works

At each update the model takes the prior belief b_t, the action a_t and the observation o_t, and writes a posterior belief b_t+1. That belief replaces the history as the working context. Each belief then gets a grade, which acts as a second RL reward alongside the task reward.

There are two kinds of grader:

• Heuristic, domain-specific. These encode what a good belief should hold. The coding example given is "shorter is better, but closer to reconstructing the git diff is also better." For Combination Lock, the grader computes useful statistics over the history and checks that they can be recovered from the belief.
• General, reconstruction-based. Use this when good heuristics are hard to define. The current policy π_θ acts as both encoder and decoder, with the belief as the code. The grade is how well π_θ can reconstruct o_t from b_t+1, conditioned on the task prompt, b_t and a_t.

For memory control, a Peak Belief length Penalty (PBP) penalizes belief length rather than reasoning length. Full details are deferred to the paper.

flowchart LR
  B[belief b_t] --> U[update]
  A[action a_t] --> U
  O[observation o_t] --> U
  U --> B2[belief b_t+1]
  B2 --> G{belief grader}
  G -->|reconstruct o_t or domain stats| R[auxiliary reward]
  B2 --> P[policy acts]
  P -->|next step| B

Evidence

• CollabBench (collaborative coding from Sweet-RL; the agent asks clarifying questions, then is scored on hidden unit tests). The reconstruction grader gives the roughly 50% gap reduction and roughly 50% fewer training steps versus no grading. Peak context tokens stay well below full context after training.
• Combination Lock (Wordle-like, up to 16 guesses; metric is average attempts, lower is better). An ungraded context-summary policy never catches up with full context over RL fine-tuning. ABBEL with the domain-knowledge grader approaches or exceeds full context. Without grading, learning is slower.
• Multi-objective QA (from MEM1; metrics are exact match and peak memory against number of objectives). ABBEL+PBP keeps performance comparable while using less memory than both MEM1 and ABBEL without PBP.

The blog gives no absolute scores. Figures are described only qualitatively or as relative percentages.

How to use it

The blog provides no code, weights or commands. Method details are in the paper: arXiv 2512.20111.

On cost, summarizing every K actions adds O(N/K) overhead over N actions. The experiments use K=1, the worst case for summary quality. The authors say that in practice K can be set near the hardware's efficient context limit.

Limits

• Combination Lock can be solved far more cheaply by other methods. The authors use it only as a testbed.
• The strongest result, matching or beating full context, depends on a hand-built domain grader. The general grader recovers only about half the gap.
• The general grader only rewards reconstructing the latest observation, not older history.
• All results come from three research environments. No production coding-assistant evaluation is shown.
• Text-only beliefs cannot capture non-textual information such as appearance. The authors flag this as an open problem.