ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences

What it claims

ScienceClaw lets an agent improve its own program across a stream of scientific tasks without changing model weights. The program is a set of Skills (strategy and recovery instructions) and typed Operators (executable subgraphs). ScienceClaw-Eval is the benchmark for this setting. It contains 23 research tasks, one per discipline, spanning the natural and social sciences. Each discipline contributes 64 IID and 64 OOD instances, and the OOD set is a different dataset from the same discipline. Code: https://github.com/beita6969/ScienceClaw.

How it works

• Typed workflows. Each workflow is a directed graph with dependencies between ports. Every port carries a schema of type, shape, unit and provenance. An edge is legal only if type and shape match and any unit conversion is explicit and recorded.
• Execution-guided editing. A frozen policy makes atomic edits: add, remove or modify a node, dependency or config. The executor reruns only the affected descendants from checkpoints. Candidate evidence is then regenerated by replay in a reset environment, because checkpoints can hold transient state.
• Deriving candidates. The repair is the shortest edit sequence between the latest replay-verified failure and the next replay-verified success.
• Control and routing edits become a Skill patch.
• New or repaired executable nodes are grouped into connected subgraphs, which become Operator candidates with pre- and postconditions.
• If there was no prior failure, only Operators are formed.
• Each Operator must pass boundary replay, meaning its isolated outputs must match the trace within numerical tolerance.
• No LLM judge ranks or selects candidates.

flowchart TD
  A[Replay-verified success] --> B{Prior verified failure?}
  B -- yes --> C[Skill patch + Operator candidates]
  B -- no --> D[Operator candidates only]
  C --> E{Boundary replay + source-task replay pass?}
  D --> E
  E -- no --> X[Discard]
  E -- yes --> F{Within budget, hard constraints hold, strictly beats incumbent MacroSR on validation set?}
  F -- no --> G[Keep current program]
  F -- yes --> H[Commit versioned update]

Acceptance rule. The source-task replay must actually use the new versioned components. The candidate must also satisfy all hard constraints, fit within a component-wise cost budget, and strictly improve MacroSR on a separate validation set.

Scoring. MacroSR is per-discipline success averaged across the 23 disciplines. An instance counts as solved only if it finishes within budget, meets the task-native acceptance rule, and violates no hard constraint (units, feasibility, convergence, reproducibility).

Evidence

The protocol is specified, but the fetched text cuts off before any results. No numbers are available here.

• Models. Qwen3.5-27B orchestrates all tasks for every system. GPT-5.4 performs the evolution step for ScienceClaw and for all baselines.
• Rounds. Snapshots are frozen after each of seven rounds, transient state is cleared, and the snapshot is evaluated on ID, OOD and replay (retention) sets. OOD results are never used to generate or select updates.
• Baselines. A frozen agent, SkillOpt, test-time tool evolution (TTE), Agentic Harness Engineering (AHE), EvoMaster and EvoScientist. Each runs its best released configuration through fixed adapters, with the same models, tools, source stream, validation data and update budget.
• Ablations. RuleEvo evolves by deterministic trace projection only. Qwen-only evolution replaces GPT-5.4 with Qwen.
• Review. Each discipline was audited by three PhD reviewers plus a domain expert. The details are truncated.

Limits

• The page text stops in the experimental setup, so gains, retention, transfer and cost figures cannot be checked here. Read the full paper before relying on the claims.
• There is one task per discipline, and each evaluation set is 64 instances per discipline. Coverage within a discipline is thin.
• The evolution step depends on a strong external model (GPT-5.4). Whether it works with the orchestrator alone depends on the Qwen-only ablation, whose result is not shown.
• Each task needs an isolated environment and a task-specific evaluator. Adopting this means building those for your own domain.