A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models

What it claims

Numeric evidence evaluation (NEE) scores groundedness by checking whether the salient financial figures in a generated claim also appear in its citations and in the source transcript. The authors argue this removes the need for expert or LLM-judge annotation. On Q4 2024 earnings call transcripts from the top 100 S&P 500 constituents, six GPT-4.1 citation baselines score 0.97–0.98 on claim–context groundedness and 0.99 on citation–context groundedness. Claim–citation groundedness is lower, at 0.907–0.959. The authors (American Express, NeurIPS 2026 attribution workshop) report that correctness on the auto-built ECTs-100 set is much weaker than groundedness. The fetched text cuts off before those numbers.

How it works

NEE

• An LLM uses structured output to extract each figure as a tuple (span, value, scalar|range, unit).
• Dates, fiscal periods, section labels, product names and version numbers are excluded deliberately, so that incidental numbers do not inflate overlap.
• A range such as "between 46% and 47%" is one atomic item. A citation containing only one endpoint gets no credit.
• Items are normalized to (type, value, unit) and must match exactly. A text with no numeric content scores 1 by definition.
• Groundedness is the fraction of figures matched, measured at three levels: claim–citation, claim–context and citation–context.

ECTs-100 construction

1. Extract the salient figures from a transcript.
2. GPT-4.1 writes one sub-question per figure, whose answer is that figure.
3. The ground-truth citation is the single transcript sentence containing the figure.
4. Each triplet is evaluated against the full transcript, so the long-context setting is kept.

Correctness metrics

• Claim correctness is precision, recall and F1 over the figures in the generated answer versus the reference answer.
• Citation correctness compares sentence identifiers rather than text.
• Scores are averaged per document, then reported as mean ± standard error across documents.

Baselines, all GPT-4.1, zero-shot, default parameters:

• Vanilla: generates claims and citations jointly.
• PostCite: writes claims first, then finds supporting sentences.
• LLMR: selects sentences first, then writes claims from those sentences only.

Each baseline runs in two citation formats. In index, sentences carry IDs. In text, the model reproduces the sentence text.

Evidence

The groundedness benchmark uses five expert-written long questions per transcript (Appendix E): outlook, risk, growth strategy, macro impact, and capital allocation. Generated claims mostly run 100–300 words.

Only claim–citation groundedness separates the methods meaningfully. LLMR-index was chosen for the correctness experiments even though LLMR-text scored higher. The reason is that sentence IDs allow citation correctness to be measured by matching identifiers.

Limits

• Truncated page. The fetched text ends partway through Section 3.2. The correctness results on ECTs-100 and the conscious-incompetence (abstention) results are not in it. Only the abstract's qualitative claim is available: correctness is notably weaker and insufficient evidence is an added challenge.
• Numeric-only proxy. Purely qualitative claims, such as strategy or risk narrative, are unverifiable and automatically score 1. This likely inflates groundedness for the long, narrative questions used.
• Matching is exact after normalization. A paraphrased or recomputed figure (e.g. a derived growth rate) counts as unsupported.
• The ground truth is generated by GPT-4.1, the same model as every baseline. Each reference answer is one figure with one citation sentence. Multi-sentence or synthesized answers are not represented.
• Narrow coverage.
• One quarter only (Q4 2024).
• 100 companies. Berkshire Hathaway is replaced by Cigna because Berkshire holds no earnings calls.
• One model, zero-shot only. No other LLMs are compared.
• Availability. The page says only that the list of companies and transcripts is "available online". It gives no dataset release, code or licence.