SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

What it claims

SCI-AGENT-EXAM (arXiv, October 2026, UCAS, ISCAS CAS and Alibaba) is a 120-task benchmark in which every task requires three deliverables: a structured final answer, executable code or scripts, and a written report. Scoring keeps execution and presentation separate. Across 10 agents run in the Claude Code harness, GPT-6 Astra leads at 80.9, 13.6 points ahead of Claude Fable 5. Every model scores higher on the LLM-judged report than on the programmatic checks. The authors also report that higher-scoring agents take fewer execution steps and regress less often.

How it works

Each task ships as a self-contained package: instruction, fixed Docker environment, visible files, deliverable spec, plus hidden checks and a hidden rubric. 156 domain experts took part, with task writers and evaluators in separate groups. A PhD-level expert builds each task, including a reference solution. That solution must score at least 90% on the human-written rubric as a calibration check. Some tasks use "trap injection": plausible distractors or misleading intermediate signals that reward shortcut solutions. Writers targeted about six hours per task. Reviewers targeted two hours and super-reviewers one hour, with an 85% agreement threshold between them.

Tasks are drawn from recent papers, competitions and datasets, and maintained open-source projects. By type, there are 28 Tool Orchestration, 25 End-to-End Analysis, 20 Code Generation, 16 Hypothesis Validation, 11 Error Diagnosis, 10 Literature Synthesis and 10 Result Interpretation. Many domain-by-type cells hold 0 to 2 tasks.

flowchart LR
  A[Agent artifacts] --> P{Required file present?}
  P -- no --> Z[Dependent points score 0]
  P -- yes --> C[Checker functions on hidden data]
  P -- yes --> J[LLM judge, binary rubric items]
  C --> S[Weighted per-type blend]
  J --> S

Programmatic checkers import the generated code or read artifacts and rerun on hidden data. They test numeric tolerances, schemas, required files and invariance under allowed perturbations. Rubric items are pass/fail and weighted. The final score is a weighted blend of the two, with weights fixed per task type. A malformed judge output is logged as a setup failure and excluded from scoring.

Evidence

Results are means ± standard error across tasks. Pass@0.95 is the share of runs scoring at least 0.95.

Claude Opus 5 has a higher Pass@0.95 than Claude Fable 5 despite a lower mean. GLM-5.3 and Qwen3.8 Max show the widest gaps between report and execution scores, about 20 points each.

A quadrant analysis measures the share of runs that are well presented but not executed, labelled TL. GPT-6 Astra has TL 0.14 and 0.83 strong overall. Claude Fable 5 has TL 0.32. Qwen3.8 Max has TL 0.40. A high TL means a report-only grader would overrate these agents.

How to use it

Code and data are at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam. Every task runs in its own Docker container with a fixed scientific stack. All agents use the same command-line harness, which builds the image, invokes the agent, runs hidden checks, applies the judge and writes an evaluation receipt. The page gives no install commands or licence.

Limits

The fetched text cuts off partway through the results. Several items are not shown:

• trajectory, efficiency and cost results;
• per-type score weights (appendix Table 5);
• contamination controls (Appendix C);
• judge calibration details (Appendix I.4).

All agents were tested in one harness, Claude Code, so scores mix model and scaffold effects. The page does not state how many runs per task back Pass@0.95. With 10 to 28 tasks per type and few per domain cell, per-type and per-domain comparisons rest on small samples. Report quality is graded by an LLM judge, though against human-written rubrics.