PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?

What it claims

PhysCount (Nanjing University and NVIDIA, arXiv October 2026) is a counting benchmark built around one property: in scenes with mirrors and glass, the number of visible appearances differs from the number of physical objects. Its 4,595 questions fall into four families. Appearance enumeration has 651 questions, entity enumeration 1,150, image-formation counting 711, and correspondence 2,083. Correspondence splits into 924 visibility-set questions, 459 overlap questions across direct, reflected and through-glass views, and 700 questions on how many appearances one entity produces. Across 40 MLLMs, the authors report that errors rise when one entity has many appearances or when its appearances cross observation modes.

How it works

Images come from Common Canvas, GPIC and other public collections. Qwen3.6-35B-A3B pre-filters them for mirror and glass scenes, and humans keep only images whose counts can be resolved by careful inspection. For each target category in an image, an annotator does four things:

1. Places a point on every visible occurrence.
2. Labels each occurrence as direct, reflected or through glass.
3. Assigns occurrences of the same real object a shared identity.
4. Writes integer-answer questions derived from that annotation.

A second annotator checks points, labels and identity groups. GPT-5.6 Sol checks whether each question is answerable from the annotation, and a third annotator corrects every flagged pair.

flowchart LR
  A[Annotated group] --> B[2nd annotator review]
  B --> C{GPT-5.6 Sol QA check}
  C -->|consistent| D[PhysCount]
  C -->|flagged| E[3rd annotator correction]
  E --> D

Scoring has two levels: accuracy per question family, and complete-group accuracy, where every question about one image and target must be correct.

Evidence

The dataset has 7,086 annotated appearances: 2,416 direct, 3,996 reflections and 674 through glass. It covers 137 normalized targets, with an average of 7.06 questions per group, and 599 of the 651 groups contain all four families. In 475 groups, at least one entity is visible only through reflection or glass. In 422 groups, at least one entity has several appearances, and in 402 of those the appearances are of different types. Answers range from 0 to 196, and appearance questions have a mean answer of 10.88.

The best of the 40 models solves 13.36% of groups completely. Higher image resolution improves access to visual evidence but adds little complete-group accuracy. Presenting a group's related questions together produces what the authors call a marked improvement in both per-question and complete-group accuracy. Scaling also helps individual answers more than full groups.

Limits

The fetched text ends at the statistics section. It has no per-model scores, no names for the 40 models, and no exact figures for the resolution and joint-questioning effects. The page gives no licence for the data or the source images. It covers only three observation modes: direct, mirror reflection and glass. The image and QA filtering relied on Qwen3.6-35B-A3B and GPT-5.6 Sol, so model-family overlap with evaluated systems should be checked. Data and details are listed at https://PhysCount.github.io.