Hey, guess what: CUES lead to inferences about typicality

What it claims

The paper (University of Edinburgh, arXiv October 2026) releases EventTyp, 2,354 (Context, TypicFollowUp, AtypicFollowUp) event tuples, and EventTypGold, an 858-tuple subset where human raters agreed the typical follow-up was typical and the atypical one atypical. Using about 900 US-based Prolific participants and 10 LLMs, it reports two findings. LLMs mostly do not agree with humans on event typicality. LLMs make human-like typicality inferences mainly when they are large models used with Chain-of-Thought prompting. The authors flag the first finding as relevant to downstream uses that rely on LLM commonsense, such as task planning and knowledge graph completion. Code and data are at https://github.com/alisonhc/heygw-eventtyp.

How it works

Corpus construction runs in six steps:

1. Contexts come from ATOMIC isBefore/isAfter pairs with at most two entities. That gives 33,924 utterances, and rule-based and LLM filtering cuts them to 12,978.
2. GPT-4o generates up to three typical follow-ups per Context, 19,827 candidates in total.
3. One author hand-annotated 9,188 pairs to derive the triggering criteria. A follow-up must be natural in dialogue, and it must be very typical but not entailed by the Context. These criteria became 27 true/false questions for GPT-4o, and a pair is rejected if it violates any one. The filter favours precision and keeps 4,155 pairs.
4. GPT-4o writes one "less typical, more specific" atypical follow-up per pair. All of them were manually validated, leaving 4,113 tuples.
5. The tuples are subsampled to one per Context (2,330), then the 24 Kravtchenko and Demberg (2015) tuples are added.
6. Humans rate every pair on a 4-point scale (Very Unlikely to Very Likely), with 3 raters per item for the commonsense experiment and 5 for the inference experiment. The agreeing tuples form EventTypGold.

LLMs are tested in three ways. Surprisal inserts each likelihood option into "In general, when [Context], it is [rating] that [Follow-up]" and picks the option with the lowest summed surprisal on the follow-up. This uses base models only, with no length normalisation, since the same follow-up is compared each time. Direct prompting and CoT prompting use the instruction-tuned versions with the human question format. The models are GPT-5.4, Llama-3.1, Olmo-3, Qwen3 and Gemma-3, two sizes each. The page reports this as 28 configurations, while 5 families × 2 sizes × 3 methods would be 30; it does not explain the gap.

Evidence

Against a human annotator on 39 sampled pairs, the LLM criteria filter scores precision 78.11, recall 69.93 and Cohen's κ = .4532. The LLM rejected 26.37% of pairs and the human 16.48%. The authors attribute the modest agreement mainly to the LLM rejecting more. The fetched text ends at §5.1, so it contains no per-model agreement numbers or inference-experiment results. The human-LLM gap and the large-model/CoT finding are stated only qualitatively.

Limits

• Most follow-ups are written by GPT-4o: 2,062 of the typical ones and 1,679 of the atypical ones. Typical follow-ups are validated by an LLM in 1,226 cases versus 1,128 by humans.
• The criteria come from one author's annotation, and the filter was checked on only 39 pairs.
• The source data contains US-specific references such as Home Depot and Thanksgiving, so only US participants were recruited. Typicality judgements may not transfer to other populations.
• The authors claim only that atypicality inferences need pragmatic reasoning. They treat typicality inferences as semantic belief updates.
• The 4-point scale has no neutral option.