fangqian616/consensus-pipeline

What it claims

The tool argues each claim of a literature review through multiple LLM agents instead of summarizing papers in one pass. The README does not report an evaluation of the debate. Its only evidence is one anecdote. In that example, an "accuracy" agent reported 10-40% error reduction for decomposition methods, and a "methodology rigor" agent flagged data leakage in the same papers. Both points reached the report. The author says that without debate only the accuracy claim would have survived, and states: "What I can't quantify yet is how much better the overall report is." The project is MIT-licensed and described as a student project.

How it works

The summary already covers search, QC and debate. These details are not in it:

• 3-layer QC. The layers are a hard filter using LLM-generated exclusion signals, then LLM domain classification, then tiering into core, method and background papers. There are no hardcoded keywords, so changing the topic requires no code changes. In the energy-economics example, 219 papers were cut to 77, a 64.8% exclusion rate.
• Department handoff. Each department sees the conclusions of the departments before it. Cross-department pairing uses a basic two-layer fallback.
• Debate length. The default is 3-8 rounds per department, with early stop when debaters converge.
• Phase 4.9. Before the debate, the pipeline fetches open-access full text in the order Unpaywall, then Semantic Scholar, then OpenAlex. It also matches any PDFs in fulltext_papers/.
• Phase 5.5 breakpoint. After the first few departments, the pipeline lists papers the debate cites that lack full text and are consensus-cited or cross-department. It pauses so you can import them. Imported and seed papers get weight=core and are always included in the report.
• Excluded from verification. Self-referential statements in the report, such as "this review included 215 papers", are not checked.
• Scoring. Claims marked ❓ unverified count toward the score. Claims marked 📖 needs-fulltext or 📭 title-only do not. The verifier is given author metadata so it can first confirm it is looking at the right paper.

flowchart TD
  C[Claim + cited abstract] --> A{Abstract NLI}
  A -->|entail| V[✅ verified]
  A -->|contradict| X[❌ contradicted]
  A -->|neutral| F{Full text available?}
  F -->|no| N[📖 needs-fulltext, excluded]
  F -->|yes| T{Full-text NLI}
  T -->|entail| V
  T -->|contradict| X
  T -->|neutral| M[⚠️ wrong-paper citation or ✂️ overstated]

Outputs go to v2_run_output/<date>_<topic>/. They include Markdown, DOCX and PDF reports, citation_verification.json, debate logs, charts and generated Python code for the key methods. Output is in Chinese by default (--lang zh); use --lang en for English.

How to use it

Requirements: git and Python 3.10+. No GPU or database is needed. Any OpenAI-compatible API works, but testing was mainly against DeepSeek.

curl -fsSL https://github.com/fangqian616/consensus-pipeline/raw/main/install.sh | bash

That one-liner pipes a remote script into your shell, so review the script before running it.

To run manually from the CLI:

git clone https://github.com/fangqian616/consensus-pipeline.git
cd consensus-pipeline
pip install -r requirements.txt
export DEEPSEEK_API_KEY="sk-your-key-here"
export DEEPSEEK_MODEL="deepseek-v4-flash"   # or deepseek-v4-pro
python run_pipeline_v2.py --topic "Your Topic" --lang en

After adding paywalled PDFs to fulltext_papers/, you can redo only the report and verification stages. This reuses the finished debate:

python run_pipeline_v2.py --topic "Your Topic" --output-dir "v2_run_output/<run-dir>" --rerun-67

The MCP server uses only the standard library. Clients listed as supported are Claude Desktop, Cursor, Codex and any MCP agent:

{"mcpServers": {"consensus-pipeline": {"command": "python", "args": ["/path/to/consensus-pipeline/mcp_server.py"]}}}

For the web UI, run streamlit run app.py, which serves on port 8501. You can edit the generated departments in the UI before the debate starts.

Limits

• The debate has no quality evaluation beyond the single anecdote above.
• Run time and cost are not given beyond "takes a while" and "inexpensive". The debate phase is the bottleneck, and more departments means more API calls. A full run is meant to be left running as a batch job.
• Local models through custom endpoints are untested.
• Some UI labels stay mixed Chinese and English in English mode. The DSH plugin is triggered with a Chinese phrase.
• Verification is only as good as the available text. Paywalled claims stay 📖 unless you supply the PDFs.