HerHealthGPT: An Evidence-Grounded Conversational AI System for Women's Reproductive Health Copy title

What it claims

HerHealthGPT is a web-based reproductive-health assistant built on Qwen3.5-9B. It decides each turn whether to ask, conclude, inform or decline, and it cites retrieved evidence. Adding relevant retrieval to a QLoRA-fine-tuned model raises concern-category accuracy from 54.1% to 59.3% on the 540-message HerHealthEval benchmark. The same change does not improve urgency assessment. The reported 85.6% clarification recall does not come from the deployed model. It comes from an offline router that borrows the zero-shot model's ask decision, because every fine-tuned configuration scored 0% clarification recall. Code is at github.com/HerHealthGPT/herhealthgpt and a prototype runs at herhealthgpt.org.

How it works

A FastAPI backend stores sessions, structured decisions and citations in PostgreSQL. The QLoRA-adapted model runs on self-hosted vLLM, and the app falls back to the hosted base model when the adapted one is unavailable. The product evidence store has 39 askable features, 19 conditions, 146 evidence records and 17 warning-sign rules (seven emergency, ten same-day urgent), built from five public sources. Retrieval uses BGE-M3 sentence-level embeddings. It returns up to eight candidates, reserves three slots for evidence statements and drops matches below cosine 0.3. NHS content is excluded from the product index for licensing reasons.

Conclusions carry one of four action codes: EMERGENCY, URGENT_SAME_DAY, ROUTINE_APPOINTMENT or SELF_CARE. A conclusion counts as premature in two cases: a more severe warning sign is one answer away, or fewer than three answers are in and an informative question remains with no warning sign triggered. A premature conclusion triggers exactly one revision.

flowchart TD
  H[History + message] --> D[Structured decision]
  R[Retrieved evidence + warning rules] --> D
  D --> P{Premature conclusion?}
  P -- yes --> V1[One revised decision]
  P -- no --> C[Validate structure, IDs, citations]
  V1 --> C
  C --> S[Store and display]

Warning signs guide the model's choice. They do not enforce escalation deterministically.

Evidence

The component tests compare six Qwen3.5-9B configurations, using greedy decoding, 4-bit loading and thinking mode disabled. The adapter was trained on 10,538 multilingual records for two epochs at LoRA rank 16 and learning rate 2×10⁻⁴. The test corpus is 49 NHS/NICHD sections, with top-3 FAISS retrieval.

• FT+RAG vs FT: +5.19 points category accuracy (95% CI [1.11, 9.26]). Against shuffled evidence the gain is +6.11 points [2.22, 10.37]. On schema-compatible messages (n=468), FT+RAG scores 68.4% against 62.4% for FT.
• Risk: FT+RAG changes risk accuracy by −0.37 points [−4.26, 3.33] and under-triage by +4.02 points [−4.86, 13.10]. It labels 1 of 18 urgent messages as urgent.
• Clarification: zero-shot reaches 85.6% recall and 86.2% specificity, with 55.4% precision. Base+RAG reaches 66.7% recall. All fine-tuned variants reach 0%.
• User study: 30 adults (25 English, 5 Arabic, 25 aged 18 to 34). 25 preferred System C, the adapted model with RAG (Wilson CI [66.4, 92.7]). Eight still hesitated to rely on its answers.

How to use it

The page gives the repository and the prototype URL but no install commands or licence. Deployment needs the Qwen3.5-9B base, a QLoRA adapter served through vLLM, BGE-M3 embeddings and PostgreSQL. The interface supports English and Arabic right-to-left layouts.

Limits

• Category accuracy measures label agreement, not factual correctness. The benchmark labels are heuristic.
• 72 of the 540 messages have labels outside the schema.
• The router was chosen after inspecting the benchmark. A live wrapper was smoke-tested only on six variations of one case, so the deployed control loop is unvalidated.
• Retrieval was precomputed, so online latency was not measured.
• The user study recorded neither per-system ratings nor exposure order, and ran no significance tests. Preference therefore does not isolate the effect of retrieval.
• The authors state that clinical validation is still required.