Human-Agent-Society/reef

What it claims

Reef is a server that sits between your agent and its model. It records each interaction, accepts a score or feedback for it, and feeds that into a recipe that updates either model weights or the agent's harness. Updated weights are synced into the serving runtime, so later requests use the new version without restarting Reef. Harness updates are offered as versioned installs.

Weight recipes need a GPU training stack (Slime and SGLang). Harness recipes need only a model API endpoint. That is the main cost fork when choosing between the two.

How it works

flowchart LR
  C[Client request<br/>x-reef-scenario] --> R[Reef endpoint]
  R --> RX[Response +<br/>record-id receipt]
  RX --> REP[/reef/report<br/>score, feedback, receipts/]
  REP --> D{Deployment recipe}
  D -->|weights| T[Training step<br/>Slime + SGLang, GPU]
  D -->|harness| H[Reefine proposes<br/>skill / rule / command]
  T --> S[Sync weights to serving]
  H --> V[Versioned install<br/>offered next session]
  S --> R
  V --> C

• Scenarios and recipes. Every request carries an x-reef-scenario header. A new scenario name creates a scenario using the deployment's configured recipe. Requests cannot select a recipe.
• Weight path (SAO recipe). Each eligible scored rollout drives a training step.
• Harness path (Reefine). The served model writes each change as one of:
• a skill
• a rules entry
• an agent command
• a pi extension

Where the host can sandbox it, the model acts as a coding agent and runs the changed harness before handing the change back. Sandboxing requires Linux with bwrap and pasta, run as a non-root user. Alternatively, REEF_PROPOSER_SANDBOX=none skips the sandbox on a trusted machine.

How to use it

Artifact and checkpoint storage requires the git-lfs system package.

Install the package:

uv venv && source .venv/bin/activate
uv pip install reef-infra

Use a source checkout (git clone, then uv pip install -e .) for the training examples. Only Reefine ships in the wheel. Other recipes live in the repo's recipes/ directory and are selected by dotted class reference.

Weight training, using the SAO example:

uv pip install -e ".[slime]" && uv pip install --no-deps --group runtime
export MODEL_PATH="Qwen/Qwen2.5-1.5B-Instruct"
export REEF_TOKEN="reef-local"
reef serve -c recipes/sao/examples/imo_answerbench/serve.yaml \
  --inference.model-path "$MODEL_PATH" \
  --reef.port "8900"

The API is shaped as follows:

• Inference. /v1/chat/completions and /v1/messages accept the provider's own request body.
• Receipt. Each response carries the receipt in the x-reef-agent-record-id header.
• Feedback. Send it to POST /reef/report with {"score": float, "feedback": text-or-object, "references": [receipt]}. The schema is validated against reef/core/reports/.

Harness refinement against a local model API:

reef serve --recipe reefine \
  --inference.upstream-url http://127.0.0.1:11434 \
  --inference.upstream-model gemma4:26b
curl -fsS -H "Content-Type: application/json" \
  -d '{"name": "my-harness"}' http://127.0.0.1:8901/reef/scenarios
curl -fsS -H "x-reef-scenario: my-harness" \
  'http://127.0.0.1:8901/reef/harness/install?adapter=pi' | bash
reef-pi evolve "when I ask you to fix a bug, reproduce it with a failing test first"

• Set REEF_UPSTREAM_API_KEY for providers that require authentication.
• State is kept under .reef/reefine/.
• Inside a session:
• /reefine <text> files a change request.
• /versions reviews versions.
• /versions <version> install installs one.
• To change the model, restart the server with a different upstream model and rerun the install command. The model ID is written into the local harness config.
• --recipe harness-evolve is the old name for the same configuration.

Limits

Authentication

• Reefine listens on 127.0.0.1:8901 without authentication unless REEF_TOKEN is set before starting.

Install-location risks

• The harness installs to ~/reef-harness/<scenario>. The page warns to keep the install root outside the agent's project directory, because a session can write there and alter what the next session runs.
• The same warning applies to the Python environment baked into reef-pi and to the Reef checkout. Code in either runs as you.
• For the codex and dsh adapters, also keep these out of /tmp and $TMPDIR, since their sandboxes can write there too.

Sandboxing

• The pre-session integrity check only holds against sessions confined to the project, /tmp, and $TMPDIR.
• The pi adapter runs commands with no sandbox at all.

Unverified in this page

• The README links to per-recipe results pages but gives no benchmark numbers itself.
• The fetched text does not state a licence.