Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents

What it claims

CLE is a benchmark for measuring whether LLM agents get better through accumulated experience, as opposed to solving isolated tasks. Each of its 120 tasks is a five-level ladder of related professional assignments. In the October 2026 v1 results, harness choice moves completion as much as model choice. The same backbone scores very differently under different agent frameworks.

How it works

Later levels depend on earlier ones through five designed forms of transferable knowledge:

• Tools and Skills: how to use tools and carry out tasks.
• Data Dependency: information and artefacts produced earlier.
• Strategy: effective approaches and recurring pitfalls.
• User Context: the user's preferences.
• Domain Knowledge: professional rules and conventions.

The environment has 23,981 real input files across 77 file extensions. It also has 19 stateful mock applications exposing 1,024 APIs, used for workflows such as merging collaborators' Git changes and scheduling meetings. Application state carries over between levels.

A multi-turn evaluation agent grades each submission. It uses tools to extract evidence from the submitted artefacts and the final environment state. It checks that evidence against 16,125 human-authored criteria, each with a single expected outcome.

flowchart TD
  A[Attempt level] --> E[Rubric evaluator]
  E -->|pass| N[Unlock next level]
  E -->|fail, K=1| X[Ladder stops]
  E -->|fail, K=2| F[Diagnostic feedback]
  F --> R[One revision]
  R --> E2[Re-evaluate]
  E2 -->|pass| N
  E2 -->|fail| X
  N --> A

Trial 1 (K=1) allows one submission per level. Trial 2 (K=2) returns rubric-based feedback after a first failure and allows one resubmission.

In the reported runs, model parameters are frozen. Only the harness state carries learning across tasks, such as memories and skills. All systems run the same fixed sequence of ladders.

CLE also supports "blocks": contiguous ladder segments that can start at any level, with the needed support files provided.

Evidence

The evaluation covers 17 configurations: 11 backbone variants across four harnesses (Claude Code, Codex, Hermes, OpenClaw). Completion is the share of all 600 levels passed under Trial 2.

The same model scores differently under Hermes and OpenClaw:

Feedback gains are small for some configurations. Hermes / Qwen3.8-Max gains +1.0 pp and Hermes / GLM 5.3 gains +1.5 pp.

How to use it

The data is on Hugging Face at huggingface.co/Continual-Learning-Exam. The code is at github.com/haoyu-zhao/Continual-Learning-Exam, and the project page is haoyu-zhao.github.io/CLE. The authors also suggest using the dataset to post-train LLMs for continual learning. The fetched text gives no install commands, licence or compute cost.

Limits

• Parametric learning untested: all results use frozen models, so fine-tuning or RL-based learning between levels is not covered.
• LLM-based grading: scoring depends on the evaluation agent. The fetched text does not name its backbone or report agreement with human graders.
• Single fixed order: a single fixed task sequence was used, so order effects are not measured.
• Truncated source: the fetched text ends partway through Section 2. Per-domain results, ablations and the authors' own limitations section were not available.