Simreal-AI/Simreal-MLBench

What it claims

This is the open, Apache-2.0 slice of SimReal's ML Benchmark: the 60-task catalog, evaluation protocol, scoring code, and client SDK. The evaluation runtime, official scoring, and full-suite admission are not in this repo — they run on SimReal's commercial engine. It is a public preview, explicitly early and expected to change, and it ships no trained results: numbers only appear once they carry full provenance.

How it works

The design turns on a trust boundary the summary names but does not draw: test labels never leave the external competition platform, so SimReal never holds the answers. An agent talks JSON tool calls through the public SDK (mleb-remote) into a restricted, forced-command SSH gateway, then a controller/worker queue runs containerized jobs, and a submission broker freezes artifacts and enforces the two-submission rule before results hit external official scoring.

flowchart TD
  A[Agent - any model/framework] -->|JSON via mleb-remote, public SDK| G[Restricted gateway<br/>forced-command SSH]
  G --> C[Controller + worker queues<br/>containerized jobs]
  C --> B[Submission broker<br/>freezes artifacts, 2-submission limit]
  B --> S[Official competition scoring<br/>external, holds labels]
  S --> R[Scoring + aggregation 100 x p squared<br/>public, recomputable]

The score 100 × p² is defined here: p is the fraction of a matching frozen reference leaderboard the agent outperforms, with half credit for ties. Source-blindness is enforced at load time — briefs are scanned for platform names, competition identities, and URLs — and every reward is probed with adversarial submissions before a task is admitted. The two submissions come with an honest real score in between, which is the point: it measures whether an agent can act on feedback, not just its one-shot performance.

How to use it

Python 3.11+ (3.12 is the locally tested version). These run against local examples with no credentials or model calls:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[test]'
mleb validate configs/catalog.json
mleb grade examples/snapshot.json examples/result.json
python scripts/verify_report.py
python -m pytest -q

With this package alone you can validate the catalog/protocol, read the scoring code, and recompute any published score from saved reference counts plus verify submission id, artifact hash, and reference snapshot hash. Actual end-to-end agent runs, official two-submission scoring, and report-eligible results require contacting business@simreal.co.

Limits

• The original competition datasets are not included — acquired separately via the data access guide.
• Source-blinding reduces but cannot eliminate the chance an agent recognizes a well-known dataset; the report flags this explicitly.
• You cannot self-host evaluation: the runtime, gateway, broker, and official-scoring route are a commercial operated layer, so any officially reported number depends on SimReal's service.
• RLVR/RSI are stated as supported directions with a proposed protocol and required trainer integration; this release includes no trained results.