AMAP-ML/LongHorizon-Harness

What it claims

The value is in the loop around the agent, not in the agent itself. The model decides what one round can do. The harness decides the next bounded step, verifies the result in the real environment, keeps progress, and continues after a failure or context refresh. It trains no model and replaces no agent. Each backend keeps its native execution loop behind a lightweight AgentAdapter. Latest version on the page is v0.1.7 (2026-08-20).

How it works

• Fresh context per step. Each step runs with a fresh context. Only output that passes the Auditor becomes task state. A rejected result is stored as evidence for the next round, not as progress.
• Executor and auditor variants. There are GUI and CLI variants of both roles.
• Per-role model settings. Each role can take its own agent, model and reasoning effort, resolved down a chain:
• gui_executor → executor → [run]
• cli_auditor → auditor → [run]

The README's example is a strong Manager and Auditor with a cheaper Executor.

• Permissions by role. Executors get workspace write access. The Manager and Auditors run read-only.
• Episode timeouts. Each timeout applies to a single role invocation, not to the whole run. On timeout, the partial trajectory is kept and the next Manager round inspects the workspace. Repeated timeouts trigger a human-review gate in the Dashboard.
• Follow-ups (v0.1.7). A follow-up message continues the same round ledger instead of replanning from scratch.

Evidence

The page says the harness was run on "hundreds" of GUI, CLI and mixed tasks. All results use Qwen 3.7-Plus as the backbone and Claude Code as the execution backend. The README states no scores. The tables are on the project website and in the arXiv paper (2608.01964). eval/ holds frozen reproduction suites for three benchmarks, including Terminal-Bench 2.1.

How to use it

Install the package:

uv tool install lh-harness            # or: pip install lh-harness

GUI tasks only: install one computer-use plugin.

• codex-computer-use works with Codex only.
• open-computer-use needs Node.js 20+ and drives both Claude Code and Codex.

lh-harness plugin install open-computer-use

Set up the project, then run from the browser or the CLI:

lh-harness init
lh-harness web --workspace-root .
lh-harness run --task "${TASK}" --agent codex
lh-harness doctor

Workbench and runs

• The workbench serves at http://127.0.0.1:8799/.
• Runs act on the launch directory by default.
• Run state goes to ./.lh-harness/runs/<run-id>/, and the final report is logs/report.json.

Configuration

• Precedence is CLI arguments, then .lh-harness/config.toml, then built-in defaults.
• Task text, run IDs and API keys cannot be set in the config file, so they never end up in a committed file.

Plugins and MCP servers

• When several plugins are installed, priority is codex-computer-use > open-computer-use > clawdcursor.
• Any MCP server can be passed with --claude-mcp-config (.mcp.json) or --codex-mcp-config (TOML).
• npm plugins live in ~/.lh-harness/ and leave ~/.claude.json and ~/.codex/config.toml untouched. The exception is codex-computer-use, which is registered in Codex's own registry.

Limits

• Platform. Only macOS is tested. Windows support is included but "not yet thoroughly tested." On Windows the harness needs a signed-in desktop session and must not run elevated.
• macOS permissions. codex-computer-use shows no permission dialog. Without manual Accessibility and Screen Recording grants, GUI calls simply fail.
• DeepSeek Harness is phase 1, developer preview:
• no GUI computer use or MCP;
• only the final answer is returned, so intermediate tool events never appear in the trajectory;
• the task text is visible in the child process's argument list while it runs.
• Workspace scope. Agents work in your real project directory unless you set --workspace.
• Eval code. The code under eval/ is a frozen compatibility copy. New integrations should use src/lh_harness/.
• Maturity. Versions 0.1.2 to 0.1.7 shipped within about two weeks, and the authors say they are "iterating rapidly."