SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots

What it claims

SIM-UT is an evaluation loop that tests production triage chatbots by having an LLM play their real user population over 10-turn conversations. The authors are from UserTrace, Cliniva, hearX Foundation, the University of Pretoria, Respira Health and Turn.io, and the paper is dated October 2026. Applied to three WhatsApp agents on the Turn.io platform, it found that both MedGemma and the frontier models de-escalated after first flagging an emergency. This happened when users downplayed symptoms without adding clinical information. Besides the bleeding-in-pregnancy case, the agents agreed that a child with an ear infection and new headache plus light sensitivity could wait until morning. They also agreed that decreased fetal movement at 32 weeks "can be normal."

How it works

There are three components: a simulated user, the production agent under test, and an evaluator LLM acting as judge. The simulated user is an orchestration of persona selection, scenario execution and turn scheduling, with claude-sonnet-4-6 writing the utterances.

The user is split into two parts. A persona holds long-term traits such as motivations and communication style. A scenario holds the short-term situation and mindset. This split follows RecAgent and SOTOPIA.

Local operators grounded the personas in samples of prior real conversations. They reviewed the synthetic dialogues over several rounds until the dialogues matched real use. The persona dimensions differ by site:

• India (Respira Health, respiratory screening): Hindi, code-mixed Hindi and English, Punjabi, Roman Hindi with Marathi words, combined with COPD, post-TB, COVID or undiagnosed symptomatic.
• Kenya (Cliniva, diabetes coaching): clinical status, structural constraints such as canteen food, gig work or limited clinic access, and stance such as social-media expectations or home remedies.
• South Africa (hearX, hearing health): teachers, principals, parents and community health workers, in isiZulu, English and Afrikaans.

Pushback, minimisation and resistance were written into the scenarios explicitly. The paper cites Ni et al. (2026) for this: simulated users are unrealistically cooperative by default.

flowchart LR
  P[Persona + scenario] --> U[Simulated user]
  U <-->|up to 10 turns| A[Agent under test]
  A --> E[Evaluator LLM]
  E --> H[Human review]
  H -->|refine rubric| E
  E --> F[Failure analysis]
  F -->|revise prompt/harness| A

Each deployment ran on two base models, to test whether the failure comes from the model or from medical fine-tuning. South Africa and Kenya used ChatGPT and MedGemma. India used Claude and MedGemma. Each chatbot went through 2 to 4 rounds, with prompt, harness and evaluation criteria updated after each round. Clinical safety and scope adherence were the risk dimensions assessed at all three sites.

Evidence

The study covers 5,000+ simulated multilingual turns. The fetched text gives the failure examples qualitatively. It reports no failure rates, no per-model comparison numbers and no before-and-after figures for the remediations.

How to use it

The fetched text gives no code, package or licence. The footnote for SIM-UT points to getusertrace.com, the site of UserTrace, a commercial company. To reproduce the method, a team needs real user conversation samples and local experts to validate the personas and scenarios.

Limits

The fetched text stops partway through Section 2.1.2, so the simulation runs, the remediation section and any limitations section are missing. The simulated user and the evaluator are both LLMs. Realism rests on validation by the local experts, not on a measured comparison with real users. The exact ChatGPT, Claude and MedGemma versions are not named in the fetched portion.