Welcome RL Environments to the hub

What it claims

There is no new repo type, registry or sign-up. An environment is an ordinary dataset repo with the rl-environment tag, listed at huggingface.co/datasets?other=rl-environment. Framework tags (harbor, verifiers, NeMo Gym, OpenEnv) do two things: they put an icon on the page and add a generated snippet to "Use this dataset". A repo can carry several framework tags. Fixes then propagate through one repo and one discussion tab: when an author fixes a bad test, every framework gets it on the next pull. This release covers tasksets only, not runtimes. Hugging Face Jobs and Sandboxes can execute workloads, but tagging launches nothing.

How to use it

To publish, add this to the dataset card YAML. The post calls this "the whole integration":

---
pretty_name: Terminal-Bench 2.0
tags:
- rl-environment
- harbor
- verifiers
---

To list a framework that isn't registered yet, open a PR to the supported-libraries list. OpenEnv already tags automatically on upload.

Harbor, oracle run. This runs the task's reference solution with no model call, which checks that the task and verifier work before you spend tokens:

uv tool install --python 3.13 'harbor==0.21.0'
harbor run \
    --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
    --dataset terminal-bench-2.1@2.1.0 \
    --include-task-name '*regex-log' \
    --agent oracle --env docker --jobs-dir results/harbor
harbor view results/harbor

Verifiers, model agent with a minimal bash harness:

uvx --python 3.13 --from 'verifiers[harbor]' eval harbor \
    --env.taskset.repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
    --env.taskset.dataset terminal-bench-2.1@2.1.0 \
    --env.taskset.tasks '["regex-log"]' \
    --env.agent.runtime.type docker \
    --env.agent.harness.id bash \
    --model "$MODEL" \
    --client.base-url "$LLM_URL"

repo must be the full Git URL, and dataset is the name@version from that repo's registry.json. A bare Hub repo ID cannot replace both.

OpenEnv, OpenCode agent. This writes the reward and trace to JSON:

pip install "openenv[harbor]==0.7.0"
openenv harbor rollout \
    --llm-url "$LLM_URL" \
    --model "$MODEL" \
    --dataset harborframework/terminal-bench-2.1 \
    --task-index 0 \
    --harness opencode \
    --sandbox docker \
    --out rollout.json

import json
from pathlib import Path
result = json.loads(Path("rollout.json").read_text())[0]
print("Reward:", result["reward"])
print("Model calls:", result["n_turns"])
print("Error:", result["error"])

Environments with stated sizes

• Multi-SWE-RL-Verified (Verifiers): 2,232 of 4,703 rows that pass gold-patch validation, covering C, Go, Java, JS, Rust and TS.
• Scale-SWE-Verified (Verifiers): 17,202 of 20,181 Python issue tasks that give a clean end-to-end reward.
• Workplace Assistant (NeMo Gym): 5 databases, 26 tools, 690 business tasks.
• Harbor-Mix: 100 hard tasks, positioned as cheaper than a full multi-benchmark sweep.
• NatureBench (Harbor): 90 tasks.
• Terminal-Lego (Harbor): StackOverflow-derived tasks, kept only if they pass a Docker round-trip.
• Reverse-Text-RL (Verifiers): a tiny task that prime-rl uses in CI to debug RL training.
• Others: CFBench (multilingual constraint following) and SysBench (multi-turn system-message following) for NeMo Gym.

Several of these show up in the filter only after maintainers merge the PRs Hugging Face opened.

Limits

• Tags declare compatibility; they don't convert files. Every listed framework must actually load the repo's layout.
• The Verifiers and OpenEnv paths need Harbor task directories. OpenEnv reads tasks/.
• The OpenEnv default routes through a temporary Gradio tunnel so the sandboxed agent can reach the model proxy.
• A reward of None means the verifier produced nothing, not that the model failed. Check error first.
• Verifiers can be narrow. NeMo Gym's Structured Outputs verifier scores schema adherence only, not factual correctness.
• Each framework gets one default snippet. Per-config snippets for multi-taskset repos and structural layout detection are planned, not shipped.
• No NeMo Gym run command is given in the post.
• Runtimes are out of scope for this release.