Skip to content

Back to the catalog

Leaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations
arxiv.orgpaper

Leaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations

This paper studies whether vision-language models can infer how long a person stays somewhere (living, staying, or leaving) from geotagged social media posts. The authors build a new corpus of geotagged tweets co-annotated by humans and LLMs. State-of-the-art models struggle even with multi-turn Chain-of-Thought prompting, often missing contextual cues and subtle hashtags while over-relying on prominent visual details.

TL;DR Introduces a human-LLM co-annotated geotagged tweet corpus showing vision-language models perform poorly at inferring stay duration.
Open original ↗
Document

What it claims

The paper releases a corpus of 4,312 geotagged tweets (text plus image, US city-level geotags) labelled with how long the author will stay in the tagged city: 0-24 hours, 1-30 days, or more than 30 days. The tweets come from GeoCOV19Tweets, a COVID-19-era collection of about 420,000 tweets. Roughly 70% of the final labels are "more than 30 days." The rest split nearly evenly between 1-30 days (13.4%) and 0-24 hours (16.3%). Corpus and code are posted at an anonymous 4open.science link.

How it works

Tweets were filtered to keep only those with a US city-level geotag and both text and images. Corporate accounts were removed through username keywords and CRAFT-based detection of poster-like images. Tweets containing explicit duration phrases such as "5 days" were also removed, so the task tests inference rather than extraction.

Labels come from two sources that are then reconciled:

  • Humans. About 2,500 Amazon Mechanical Turk workers took part, with a 95% approval-rate floor, a 5-second minimum per HIT, and five annotations per HIT. The lowest-competence worker (scored by MACE) was dropped repeatedly until Krippendorff's α reached 0.6. Final average α is 0.68, with about 2.5 annotations kept per tweet and about 30% of raw annotations discarded.
  • LLMs. Six models were used: Qwen 2.5, GPT5-mini, Grok 4, Claude 3 Haiku, Gemini 2.5 Flash and Gemini 2.5 Pro. Qwen 2.5 and GPT5-mini scored κ below 0.2 and were dropped. MACE then aggregated the remaining four.

Humans and LLMs agreed on only about 36% of tweets (the diagonal of Table II: 2.00 + 2.65 + 31.69). The largest disagreement is the LLM saying "more than 30 days" where humans said "0-24 hours," at 32.23%. Conflicts were resolved by cause:

  • Quarantine misreads (R1) and city-level errors (R2). The shorter label was rejected.
  • Unclear intent (R3) and unclear residency (R4). The tweet was discarded.

Evidence

The experimental setup is a stratified 80/20 split. Only the first image of each tweet is used, and no user profile information is used.

  • Fine-tuned baselines. RoBERTa-Large (text) and ViT-Large/16 (image), run text-only, image-only, and fused by concatenation, addition or element-wise product, each with a two-layer classifier.
  • LLMs and VLMs. Evaluated zero-shot, without fine-tuning.

The fetched text ends before the results tables. No accuracy or F1 numbers are available here. The intro states only that multi-turn Chain-of-Thought and reasoning models "significantly improve performance" yet still fail on ambiguous and culturally nuanced cues.

Limits

  • Class skew and resolution rule. The 70% "more than 30 days" share, together with a rule that rejects the shorter label for R1 and R2 conflicts, may push labels toward long stays.
  • Pandemic data. All tweets come from the pandemic period. The authors acknowledge behavioural bias but assert that the reasoning generalises, without testing this.
  • US only. Geotags are restricted to US cities.
  • Kappa reporting. Table I is captioned Cohen's kappa while the text says Fleiss' kappa. Its interpretation column labels κ of 0.17 as "moderate" and 0.53 as "substantial," which conflicts with the 0.2 exclusion threshold used.
  • Licence. No licence for the corpus is stated. Redistributing tweet content may also depend on X's terms.

Organisations named here

  • arXiv

Technologies named here

  • Vision language model
  • LLM

Techniques named here

  • Human-LLM co-annotation
  • Multi-turn Chain-of-Thought prompting

Related in research