FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents

What it claims

FinAutoRubric generates a grading rubric for each financial-research query. Experts never author individual items. They write reusable guidance once, and agents plus code turn it into criteria with researched expected values, fixed as of a stated information cutoff. The authors position it as the only finance benchmark that is open-ended, auto-generated, on-demand (a new query or cutoff needs no re-authoring) and accepts custom institutional guidance. They compare it against 12 others, including BigFinanceBench, FrontierFinance and Finance Agent Benchmark. Their figure reads as follows:

How it works

Expert guidance acts in two ways: as prompt instructions to every agent, and as rules an orchestrator enforces in code. When an output breaks a rule, the orchestrator rejects it and returns it for repair.

1. Task Bank (Stage 1). Each analyst query is reduced to a value-free task signature. The signatures are embedded and clustered. Models from two different families write atomic criteria, each tagged with the expert principles it realises. Experts only screen the result and remove duplicates. The frozen bank holds 200 tasks.
2. Contextualization (Stage 2). This runs before any retrieval, so no retrieved value can shape the task definition. - An agent splits the query into a task signature plus eight typed binding arrays: entities, periods, metrics, assumptions, deliverables, output constraints, convention topics and populations. - Every binding must copy a span of the query verbatim. - The agent matches the query to a bank task only on a complete and unique fit. Otherwise it writes criteria from the principles alone. - Code writes the cutoff into the context as an inclusive date.
3. Rubric generation (Stage 3). - Point budget. Before any research, code splits 100 points equally across the query's obligations and freezes that split. The authors' reason is that weighting by importance up front would bias the search toward what the model already expects to matter. After research, a later composition step may move points between obligations. - Writer. The writer uses web search and Python. It never sees a reference answer or a response under evaluation. - Every expected value must come from a cited source or an executed script. A value that cannot be grounded is left unresolved. - Numeric targets carry a tolerance. - If the evidence supports several conventions, each gets its own accepted value. An answer can follow one route but cannot mix routes. - Reviewer. A reviewer from a different model family, in its own isolated sandbox, reopens every source, checks it against the cutoff and re-derives each value.

flowchart TD
  Q[Contextualized query + frozen budget] --> W[Writer: search + Python]
  W --> V{Code validator}
  V -- fail --> W
  V -- pass --> R{Cross-model reviewer re-derives values}
  R -- fail --> W
  R -- pass --> G[Global loop: whole-rubric review]
  G -- gates fail --> H[Escalate to human]
  G -- pass --> Rq[Final rubric]

Evidence

According to the abstract, on the three expert-authored benchmarks named above, the generated rubrics:

• track expert scoring as closely as the strongest generator evaluated,
• state the expert rubric's expected value for more criteria than any generator evaluated,
• agree with human grading of responses,
• are preferred by in-house analysts in a blind review.

The released benchmark scores six solvers. The authors also report that rubrics produced with an earlier model generation still leave headroom for a later one. The fetched text cuts off before the results section, so no metrics, baselines or solver scores are available here.

Limits

• Not yet released. Code and data "will be publicly released soon."
• Truncated source. The quantitative claims above cannot be checked from this page.
• Light expert involvement. Experts screen the Task Bank only at a high level, and models write every criterion. Rubric quality therefore rests on the guidance, the cross-model review and human escalation, not on per-item expert review.
• Shared data access required. The writer must only use sources at or before the cutoff that the evaluated agents can also reach, such as data connectors exposed to both. Adopting it with internal or licensed data means providing that shared access.
• Equal initial budget. Points start equal across obligations and can only be rebalanced afterwards. Check whether this matches your institution's weighting.