On the possibilities and limitations of LLM-based multi-agent systems for multi-criteria decision-making in healthcare operations management

What it claims

LLM-PADA (Preference Assessment and Decision Analysis for LLM-based agents) is a four-step audit for criterion weights produced by role-playing LLM agents before those weights enter a weighted-sum optimization model. On two healthcare tasks run with locally executed Qwen3.5-family models, the paper reports three findings. Robust weighted-sum models pick a different decision than mean-weight models in 10 of 16 outpatient comparisons and 5 of 6 emergency department (ED) comparisons. LLM deliberation can weaken context sensitivity and erase or reverse role differences, while plain averaging keeps them. In pairwise choices, outcome differences improve prediction beyond presentation order for only 2 of 6 model-role combinations.

How it works

The framework has four steps: obtain preferences, assess their reliability, build uncertainty sets for the weights, and compare mean-weight against robust weighted-sum decisions.

flowchart LR
  A[Role-prompted agents] --> B[Explicit: 1-9 ratings]
  A --> C[Implicit: pairwise choices]
  B --> D[Reliability checks]
  C --> D
  D --> E[Weight uncertainty sets]
  E --> F[Mean-weight vs robust McRow]

Explicit elicitation (outpatient scheduling). Intake, provider and scheduler agents rate patient waiting time, provider idle time and provider overtime on a 1–9 scale. Prompts describe different clinic conditions. Ratings are then revised over three rounds under one of three protocols:

• numerical, where agents exchange scores;
• argumentative, where agents also exchange justifications and objections;
• mechanical averaging, which makes no further LLM calls.

Implicit inference (ED planning). Patient, physician and operations agents choose between pairs of nondominated ED configurations. The configurations route low-acuity patients to a minor injuries unit and come from the authors' earlier discrete-event simulation study (Boresta et al.). Each option is described by average waiting time and weekly unit opening hours. A logit choice model of the Bradley–Terry/random-utility type includes a term for which option is shown first. It is evaluated on held-out comparison pairs.

Robust step. Weighted-sum models are built over the weight uncertainty sets using the multiexpert multicriteria robust weighted-sum (McRow) approach of Hu and Mehrotra.

Evidence

Explicit ratings respond to clinic operating conditions and differ by role. Both deliberation protocols can flatten or invert those differences. Mechanical averaging preserves role differentiation.

Pairwise choices depend largely on presentation order. Adding outcome information helps out-of-sample prediction in only 2 of the 6 model-role combinations.

Mean-weight and robust models disagree in 10/16 outpatient and 5/6 ED cases. In this setup, the choice of aggregation usually changes the selected schedule or configuration.

Limits

"Reliability" here means internal checks only: context sensitivity, role differentiation, and stability under changes in procedure and presentation order. The paper does not compare agent preferences with real patients, providers or managers, and states that no external reference exists.

Experiments cover two healthcare settings and locally run Qwen3.5 models. Proprietary LLMs are claimed to fit the framework but are not tested. The fetched text ends partway through related work, so model sizes, prompt details, sample counts and the uncertainty-set construction are not available here.