LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents

What it claims

LiveSearch, from StepFun and collaborators (arXiv, 27 October 2026), stores each financial question as a triple: frozen question text, an executable gold program, and a fixed matching rule. The authors argue that three weaknesses of existing finance agent benchmarks come from one design choice, storing a question as a (text, answer) pair. Those weaknesses are costly curation, stale answers, and evidence drawn from web or free-tier sources. LiveSearch scores agents that use ordinary web search and page retrieval against answers recomputed from professional data. Even with tolerance-aware matching, reusing a gold snapshot from the previous day fails on a majority of questions.

How it works

The gold program is a short typed DAG. It makes one or more tool calls, then runs a declarative extractor with no branches or expressions, so replay is deterministic. An extractor names a response path, a selection rule, a field, an optional aggregate, and a format. The selection mode (latest, snapshot, match, minute, all) encodes the time anchor. For example, "latest" over trade_date rolls forward each trading day. Each record also has a refresh_class, ranging from daily to static.

Matching rules are exact, approx, or semantic. Units are normalized first, and a sign mismatch fails under every rule. The LLM proposes a rule, and the pipeline clamps it mechanically. Exact is allowed only when nothing marks the value as derived, such as a percentage unit or lossy rendering. A gate rejects any question text containing an absolute date, Chinese forms included. Concrete date windows live only in program arguments, which the daily refresh rewrites.

Construction is answer-first, in two stages.

1. Probe. A tool counts as usable for an asset class when a strict majority of sampled entities return data. Asset classes are A-share, Hong Kong and US equities, indices, and funds.
2. Coverage. For each usable tool, one entity is drawn at each of eleven web-visibility levels (0 to 10). This spreads the pool from obscure to well-known instruments.
3. Enumerate and verbalize. Every extractor that returns a non-null value on an observed response becomes a candidate. Mechanical rules filter candidates, then an LLM keeps or rejects them and writes the question text.
4. Stage 2, with no LLM. Signatures that every model solves are dropped, and templates are re-instantiated with fresh entities. Each day the programs are re-dated and re-executed.

Each refresh writes an immutable gold_answers_<date>.jsonl. It records the formatted answer, the raw value, the observed period, and the arguments sent. Question IDs are content hashes of the tool, entity, non-date arguments, and extractor, so a question keeps its identity across regenerations. The evaluator fails the run rather than falling back to a stale value if the snapshot misses any served qid, has unknown qids, or has empty or error answers.

Evidence

The data service exposes 96 tools across 9 services over the Model Context Protocol. Search tools, intraday tools, and utilities such as the system clock never define a gold program. Answers are judged by an LLM with a binary verdict. The judge comes from a model family absent from the leaderboard, and numeric tolerance follows FinSearchComp. The fetched text cuts off before the results tables. Model names, question counts, and per-category numbers are not visible.

How to use it

A same-date leaderboard comparison requires identical qid sets and gold dates. A longitudinal audit fixes the qids and tracks each date's frozen gold value. The fetched text gives no repository, licence, or access terms for the benchmark or for the "production financial data service" it runs on. Running fresh refreshes likely depends on access to that service.

Limits

Scope covers listed equities, funds, indices, macro series, market microstructure, and corporate events. Derivatives, FX, commodities, insurance, and structured credit are out of scope. Pool selection drops signatures that every model solves, so scores are relative to a pool filtered against the tested models. The authors compare against FinSearchComp, which has 635 questions and refreshes only its time-sensitive tier. They position LiveSearch as the only benchmark that is both time-sensitive and grounded in professional sources.