Ethan Walkrman (@ethanwalkerman) on X

What it claims

Harness2, from Google Cloud AI Research and Northwestern University, uses an Editor model to rewrite an agent's harness between runs while the worker model's weights stay frozen. This description comes from an X thread by @ethanwalkerman, not from the paper itself. The thread reports two things the summary omits. First, the default Editor model is Anthropic's Claude Opus 4.8, even though the worker under test is Gemini 3.5 Flash. Second, the optimized GLM-5 run took 15% fewer steps than its baseline.

How it works

The harness is split into 8 text components stored on disk: system prompt, guardrails, memory, skills, tool guidance, executable scripts, subagent routing and middleware plugins. The Editor can write new Python scripts, add middleware filters or change subagent routing, not only edit the prompt.

When a baseline run struggles, the Editor proposes two variants. One targets general planning and the other targets domain rules. Both run in the live environment. The Editor compares their execution trajectories side by side and merges only the edits that unblocked the workflow.

flowchart TD
  A[Baseline run struggles] --> B[Planning variant]
  A --> C[Domain-rule variant]
  B --> D[Compare trajectories]
  C --> D
  D -->|edit unblocked| E[Merge edit]
  D -->|no gain| F[Discard]
  E --> G[Lesson to EXPERIENCE.md]
  G --> H[Reset to Base Harness]
  H --> A

Lessons go into EXPERIENCE.md (indexed by INDEX.md) in a fixed 4-part format: cue, lesson, evidence from logs, and confidence. Before each new task, the environment resets to the clean Base Harness. Only the text lessons carry over, so code changes do not accumulate across tasks. Because every component is plain text or JSON, the harness state can be diffed and rolled back with git.

The thread cites one worked example from the harness2-demo repo. In an accounting task, the baseline script used if amount > 0, which dropped negative refunds. A probe without that filter made the total match the credit entries, and that lesson was saved.

Evidence

On WorkBuddy-Bench, GLM-5 scored 64.2 at baseline. After one optimization round (k=1) it scored 75.7 and took 15% fewer steps. Unoptimized Gemini 3.5 Flash scored 72.4. The thread gives no other benchmarks, variance figures or ablations.

Limits

• Latency: each task needs 20 test rollouts, which rules out real-time use.
• Experience poisoning: a shortcut that works on one task can be saved to EXPERIENCE.md and degrade later tasks.
• No sandbox by default: in the open-source repo, the Codex execution sandbox is disabled and internet access is unrestricted, so generated scripts run on the host.
• Benchmark leakage: the README warns that without firewall rules, the agent may write a scraper during a probe to look up public benchmark answers.

The thread's author also claims that PAC learning theory predicts unconstrained self-modifying agents will weaken their own safety checks over time. The thread cites no source for this.

The repository link in the thread is truncated (github.com/google-researc…), so the exact repo path, licence and install steps are not given.