Statistical Detection of Positional Bias in LLM-Based Recommender Systems: Methodologies and Empirical Evidence

What it claims

Carlos Peláez-González, Francisco Herrera and Andrés Herrera-Poyatos (University of Granada, October 2026) test whether a top-1 LLM recommender's choice depends on where each candidate sits in the prompt. They evaluate eight LLMs, and all eight show statistically significant positional bias. The paper splits that bias into two kinds. In "soft" bias, position decides among items the model treats as comparably relevant. In "strong" bias, position pushes the pick to a clearly less relevant item. Counterfactual analyses show the strong component is not explained by the identifiers assigned to items. For six of the eight models, it is also not explained by stochastic decoding alone.

How it works

POB-ICM treats input position and selected item as a contingency table. It then runs a query-level permutation test of independence, which supports both detection and interpretation of where the bias sits.

POB-UNBP uses a stronger null: with no positional bias, the probability of picking the item at each position is uniform in expectation over queries. Subsection 3.4 proves that this follows from absence of bias. The stronger null needs fewer samples than the contingency test and remains valid under stochastic decoding.

The three metrics measure different things:

• Permutation sensitivity is how often the decision changes when the order changes. It is an upper bound on positional bias.
• Positional bias is the total variation distance between the uniform distribution and the distribution of selected positions.
• Strong positional bias uses query-dependent "relevance sets," the candidates the model treats as comparably relevant. It counts selections that position pushes outside that set.

The testbed is a RAG pipeline. An embedding retriever passes the top-k items from a real product catalog into a fixed prompt template. The query set is 2,000 template-generated queries with 20 permutations each. Robustness is checked by varying candidate-set size k and temperature t.

Evidence

The fetched text ends in the related-work section, so it contains no per-model numbers, metric values or p-values. It also does not name the eight models. Only the qualitative outcomes are available: the null is rejected for every model, the bias is model-dependent, and certain positions show reliably elevated selection probability.

How to use it

The code is at https://github.com/GoComputing/Positional-Bias.git. The page gives no install or run instructions. The authors suggest order randomization and standardized prompting as governance controls for deployed systems.

Limits

The scope is top-1, single-turn recommendation with one fixed prompt template and synthetic queries. The paper measures bias and does not propose or test mitigation. The authors leave formal statistical detection of soft and strong bias as separate components to future work. For two of the eight models, stochastic decoding may account for the strong component.