Open-sourcing AstaBrief, the fast report-generation model in Asta

What it claims

AstaBrief 8B (from Qwen3-8B) writes the whole cited report in one pass from a query plus retrieved snippets. It skips the snippet-summarization and clustering stages, and the section-by-section writing, used by Asta's Claude-powered Thinking mode. Across the full Asta pipeline, Fast mode averages 51.1 s per report versus 178.5 s for Thinking mode. Ai2 reports this single-pass design cost no measurable quality. The main lesson for anyone building something similar: one filter, dropping synthetic reports with low citation density, produced the biggest grounding gains. More aggressive filtering, combinations of filters and learning-rate sweeps added nothing meaningful.

How it works

1. Query pool. Real Asta/ScholarQA user logs, with beta-tester and bot traffic removed and too-short queries dropped. An LLM pass removed non-English, non-scientific and PII-containing prompts. 90K queries remained.
2. SFT targets. The multi-step ScholarQA pipeline generated reports, backed by Claude 3.5 Sonnet, Claude 3.7 Sonnet, o3, o4-mini and GPT-4.1. After filtering, 47K examples remained.
3. DPO pairs. These came from queries held out of SFT and were trained on top of the filtered SFT checkpoint.

flowchart LR
  Q[Held-out query] --> A[ScholarQA + Claude 3.5/3.7]
  Q --> R[Same retrieved excerpts]
  R --> B[o3 / o4-mini / DeepSeek-V3 / R1]
  A --> J{GPT-4.1 and DeepSeek-R1 agree?}
  B --> J
  J -- yes --> K[Keep pair]
  J -- no --> D[Discard]

The judges had 95% agreement with human preferences. Only pairs where both judges agreed were kept, which yielded about 6K pairs.

The four filters tested on SFT data:

• Output/input token ratio: a high ratio means much text from little evidence.
• Mean retrieval relevance of cited papers.
• Citation density: the share of statements with at least one citation. This was the winner.
• Citation diversity: papers cited divided by papers retrieved.

The authors chose SFT+DPO over RL (as used in their DR Tulu work) because RL is unstable, expensive and harder to debug.

Evidence

• Primary eval: SQABench-CS2, 200 user-written CS questions. Metrics are rubric score, answer precision, citation precision and citation recall. The text gives no per-metric numbers; they appear only in charts not captured here. Unfiltered SFT trailed the Claude pipeline on answer precision and citation quality. Filtering plus DPO brought it "within range" of the Claude pipeline and DR Tulu.
• Secondary eval: DeepScholarBench (63 queries from recent arXiv papers). Its scores are not comparable to SQABench-CS2.
• Human study: 14 questions from 3 researchers, ranking 3 systems. DR Tulu won overall preference. Two of the three researchers preferred AstaBrief on citation accuracy.
• Production usage: 374 users tried Fast mode.
• 29.1% used it on two or more days.
• Users averaged 3.67 report threads.
• 23% never switched back to Thinking mode; another 18% alternated between the two, using Fast for about 40% of threads.
• Positive-feedback rate was 84.2% for Fast versus 85.2% for Thinking, on sparse feedback.

How to use it

Weights and training data are on Hugging Face. Ai2 also releases an example workflow for generating reports from your own PDFs, intended for local or behind-firewall deployment where queries reveal unpublished work. The page gives no commands, licence or hardware requirements. The model expects a query plus retrieved excerpts, so you supply the retrieval yourself. It was designed as one component of the Asta pipeline, not as a standalone model.

Limits

• Training and evaluation were done in 2025 against that year's frontier models and have not been rerun against current models. The authors present the results as validation of the recipe, not of where the model ranks today.
• The LLM-judged pairwise win rates against Thinking mode are favourable to AstaBrief by construction: its DPO stage optimized for exactly that pairwise ranking, while DR Tulu's did not.
• The human study is tiny (14 questions, 3 raters).
• Metrics check whether a citation supports a claim. They do not check whether the model overstates its sources, for example by generalizing from a sample to a population, shifting past-tense findings to present tense, or turning descriptive results into recommendations. The authors flag this as unmeasured.
• The primary benchmark covers computer science only.