JFrog: Keep your Agents Under Control with agent-belt

What it is

agent-belt — an Apache-2.0, JFrog-built CLI eval framework, open-sourced (per the post, "today"). It runs a coding-agent CLI you already installed and authed as a black-box subprocess against a real workspace with your MCP servers and skills wired up. Seven bundled agents: claude-code, codex, copilot, cursor, gemini, goose, opencode. Ships four exporters, sandbox providers (host/docker), and one plugin contract across every extension point.

Getting started

pip install agent-belt
belt quickstart claude-code

The post claims "minutes to a verdict" assuming the target CLI is already installed and authed.

How scenarios work

Scenarios are version-controlled JSON with multi-turn turns. The concrete assertion vocabulary the summary doesn't spell out:

• tools_invoked_any — AND-of-OR: a list of inner lists; each inner list is "any of these" (absorbs naming variance like query_events vs query_usage), and every category must be touched. Uses full MCP tool names, e.g. mcp__billing__query_events.
• contains — substring checks on a turn's output.
• max_llm_turns — turn budget (12 in the example).
• files_modified_exact — an allow-set; any extra file modified fails the turn.
• git_diff_contains — asserts diff substance, not just that a file was touched.
• state_expect: { capture_git_diff: true } — captures the diff as state.
• flags — per-turn CLI flags passed through, e.g. ["--permission-mode", "acceptEdits"].
• llm_scorer_instruction — per-scenario rubric biasing the default trajectory and response_quality dimensions. The post positions this as where negative constraints live that rules can't express ("don't reach for issue_credit", "don't promise a refund"), including hard-fail-on-tool-call semantics.

Two scoring modes combine (rules + LLM); the post also names multi-judge consensus as a scoring feature (text cuts off before detailing it).

Why it differs

The post's explicit contrast: LLM-eval frameworks score a model on a prompt; observability platforms score a function you wrapped in their SDK; trajectory frameworks build their own agent loop (so the unit under test is theirs); coding-agent benchmarks freeze a dataset. agent-belt instead drives the shipped CLI unchanged, so identical scenarios and scoring run across agents that share nothing — "one diff to compare them on your repo on the same day."

Caveats

The post is upfront: it "won't stop the agent from doing the wrong thing" — it's detection, not enforcement. The fetched text is truncated mid-sentence at the scoring-modes section, so the rules/LLM combination and multi-judge details aren't fully given here.