From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries

What it claims

Explicit grounding rules written into the prompt make LLM summaries of structured menstrual-health records hedge more on uncertain data. The authors report this across GPT-4o, Qwen2.5-14B-Instruct and Phi-4-mini (4B), over 500 records and 4,500 summaries. The paper's main reusable piece is a five-part taxonomy of "semantic drift", each part with its own automatic metric: faithfulness violation, epistemic misalignment, diagnostic inference overreach, omission drift and urgency misalignment.

How it works

The data is semi-synthetic. A causal DAG generates 500 single-day records in a fixed order:

1. Cycle and luteal-phase length are calibrated to population data and set the phase.
2. Phase drives sleep.
3. Phase and sleep drive resting heart rate and the HRV trend.
4. SpO2 is sampled independently.
5. Symptoms are generated downstream of these states.

Five user profiles, from minimal_tracker to advanced_wearable, then mask fields to mimic real missingness. A confidence score derived from completeness and phase clarity is attached as uncertainty metadata. The pre-masking record is kept as ground truth.

Each record is summarized under three regimes:

• Base: the model summarizes the record with minimal instruction.
• Constrained: the prompt adds rules for factual grounding, uncertainty handling, nuance preservation and diagnostic limits. There is no decoding-time control.
• RAG-Constrained: the same rules plus retrieved lay definitions from README, a set of over 50,000 medical-term and lay-definition pairs.

Each metric is scored so that 0 means aligned:

• Faithfulness: atomic claims are checked against the textualized record with pritamdeka/PubMedBERT-MNLI-MedNLI. Only an "entailment" label counts as supported.
• Epistemic: the metric is the share of definitive claims among uncertainty-sensitive claims. A record counts as uncertain if its confidence is below 0.60 or any field is missing. A claim is uncertainty-sensitive if it uses cycle or interpretation keywords, and definitive if it has no hedge words such as "may".
• Diagnostic: scispaCy en_ner_bc5cdr_md with medSpaCy negation finds disease mentions that are not negated and not in the record. A corrected variant excludes "pregnancy" and "ectopic pregnancy".
• Omission: reversed NLI tests whether the summary entails a statement for each non-null salient field.
• Urgency: Claude-3.5-Haiku at temperature 0 rates tone from reassuring to alarming on a 0 to 1 scale.

Readability is measured with FKGL as the primary endpoint, plus SMOG, ARI, Dale-Chall, MTLD and parse depth. Jargon is counted as UMLS-linked entities per 100 words.

Evidence

The fetched text stops partway through the statistical procedure, so no result tables are available. The stated design uses paired Wilcoxon signed-rank tests with paired Cohen's d and rank-biserial effect sizes, run within each model across regimes. The only results available are the abstract's directional claims:

• Uncertainty preservation improves under the constrained regimes.
• Faithfulness is kept or improved once it is judged against retrieved evidence.
• The effect on omissions depends on the model.
• Reading difficulty and medical terminology both go up.

Limits

• All data is semi-synthetic, and records are single days rather than real longitudinal series.
• The epistemic metric uses keyword and hedge-word heuristics.
• The urgency score rates tone only. It does not judge whether that tone is clinically warranted.
• The diagnostic metric needed a manual correction for pregnancy mentions.
• Constraints are applied only through prompting.
• The authors call their configuration findings "preliminary".
• The related-work section notes that NLI degrades under domain shift and that entailment is not the same as factual support. Both caveats apply to the faithfulness and omission scores.