Skip to content

Back to the catalog

SeClaw
github.comrepo

SeClaw

SeClaw is a benchmark framework for evaluating security risks of autonomous LLM agents, combining spec-driven synthesis of security tasks (covering resource, task, environment, and intrinsic-agent risk categories) with execution-based evaluation in a Docker sandbox. The repo ships SeClaw-Bench, a public dataset of 150 OpenClaw task directories, plus batch-execution scripts that record full agent trajectories (tool calls, file ops, outputs) and grade them against a multi-dimensional risk rubric covering information leakage, privilege misuse, content safety, and compliance risks, rather than only judging final responses.

TL;DR A Docker-based benchmark and 150-task dataset (SeClaw-Bench) for trajectory-aware security evaluation of autonomous LLM agents.
Open original ↗
Document

What it is

SeClaw (arXiv:2606.02302) is a security-evaluation framework for autonomous LLM agents, released with SeClaw-Bench — 150 OpenClaw task directories under tasks/openclaw/ plus a Docker batch runner and grading scripts. The repo is a runnable evaluation harness, not just a paper.

How it works

Two stages. Task synthesis: starting from a risk taxonomy (resource, task, environment, intrinsic risks), human experts and Claude Code co-write a structured spec (objective, threat scenario, required artifacts, constraints, acceptance criteria), abstract it into reusable constraints, then into environment-independent multi-agent prototypes, which specialized agents instantiate into executable settings with workspaces, mock services, MCP tools, and local files. A human–multi-agent QA loop reviews each task and iteratively refines failures. Evaluation: each task runs in an isolated Docker container that records the full trajectory — prompts, tool invocations, file operations, intermediate observations, final outputs — normalized into structured logs and scored against a rubric covering information leakage, privilege misuse, content safety, and ethics/compliance. The distinguishing point is that scoring examines the execution process, not only the final response.

Preconditions / Affected

Requires uv, Docker, and an OpenAI-compatible model provider (configured via .env and docker_models_config.yaml). The agent harness is OpenClaw, pulled as ghcr.io/openclaw/openclaw:main. Broader model families (Qwen, Kimi, GPT, Gemini) and other harnesses (Claude Code) are listed as roadmap, not shipped results — so today you evaluate OpenClaw agents against your chosen provider models.

What to do

Setup:

uv sync
docker pull ghcr.io/openclaw/openclaw:main
cp .env.example .env
cp docker_models_config.example.yaml docker_models_config.yaml

Edit .env and docker_models_config.yaml for your provider, then run the batch:

./scripts/batch_execute.sh \
  --backend docker \
  --tasks-jsonl batch_inputs/task_list.jsonl \
  --models-config docker_models_config.yaml \
  --docker-concurrency 2 \
  --batch-logs batch_logs \
  --batch-name docker_eval

Output lands in batch_logs/{batch_name} with aggregate scores, a report, normalized traces, transcripts, grader outputs, and execution metadata. Grading uses a separate judge model (judge_models_config.example.yaml); see docs/ for Docker, dataset, task, and grading guides.

Caveats

No published cross-model results yet — the paper's comparative benchmarking across Qwen/Kimi/GPT/Gemini is stated as planned. Evaluator calibration and trajectory-level interpretability are named by the authors as ongoing improvement areas, so treat rubric scores as directional. Grading depends on a judge model you supply, inheriting that model's biases.

Technologies named here

Techniques named here

  • Spec-driven security task synthesis

Related in AI Security