Benchmarking Answer-Grounding Consistency in Remote Sensing Visual Question Answering

What it claims

Grounded RS-VQA models are usually scored on answer accuracy and box IoU separately. Neither score checks whether the boxes a model outputs actually back the answer it gave. The paper's example: grounding 9 of 10 storage tanks gives a high localization F1 but does not support the answer "ten". AESP scores that consistency directly. Across seven open models, consistency is poorly predicted by accuracy or box precision, and exact counting is the weakest case for every model.

How it works

flowchart TD
  R[Model response + boxes] --> P{Already Yes/No or integer?}
  P -- yes --> A[Answer â]
  P -- no --> L[Qwen3-14B extracts from question + response]
  L --> V{Ambiguous / contradictory?}
  V -- yes --> Z[s = 0]
  V -- no --> A
  B[Predicted regions] --> C[Count GT instances whose box centre falls in any region: n_c]
  A --> T{Question type}
  C --> T
  T --> E[Existence: Yes needs n_c≥1, No needs n_c=0]
  T --> K[Counting: n_c = â exactly]
  T --> M[Comparison: relation on n_A, n_B matches â]

• Coverage: an annotated instance counts once if its ground-truth box centre lies inside any predicted region. No IoU is involved.
• Support is judged against the model's own answer, not the reference. A wrong count can still be "supported" if the boxes match it.
• Comparison: for "more than", Yes needs n_A > n_B and No needs n_A ≤ n_B. "Fewer than" uses < and ≥. Equality uses = and ≠. Neither class needs full coverage.
• Metrics:
• ES_overall: the share of samples with s = 1.
• ES_correct: s = 1 among correctly answered samples.
• Joint Success: correct and supported.
• Localization: micro P/R/F1 with one-to-one matching at IoU ≥ 0.5.

Dataset (G-RSVQA): images and boxes come from DIOR, NWPU VHR-10 and SODA-A, with questions and answers generated by rules. It has 8,592 existence, 4,275 counting and 2,160 comparison questions. Counts range from 0 to more than 10. Every instance of each target class is annotated. Accuracy is reported on all 15,027 samples. Support and localization use a 9,699-sample subset, which excludes the 5,328 existence questions whose answer is No.

Evidence

Table 1 (percentages; evaluation subset except accuracy):

• Accuracy does not predict support. InternVL3-8B and Qwen3-VL-8B have near-equal accuracy, but ES_overall is 50.78 vs 80.11.
• Box precision does not predict support either. GeoGround has the best precision but only 50.82 ES_overall.
• Counting is the weak spot for every model. ES_overall on counting is lower than on existence for all seven. InternVL3-8B scores 72.13 on comparison but 30.20 on counting. Qwen3-VL-8B leads on counting with 72.37 ES_overall but 50.69 JS. The gap is supported-but-wrong counts.
• Extractor validation: on 400 human-labelled responses, Qwen3-14B extracts answers with 98.50% accuracy (counting 96.23). Qwen3-8B scores 95.75 and Mistral-NeMo-12B 92.00.

How to use it

Code: https://github.com/pingggg516/AESP-Evaluation. The page gives no install commands, data licence or release terms.

Models were run from public weights with no training on G-RSVQA. GeoGround used its task-specific LoRA weights. Prompts and coordinate formats were adapted per model. Scoring requires Qwen3-14B for answer extraction, plus the object-level annotations.

Limits

• Narrow scope. It covers only three rule-generated question types from three detection datasets, with no free-form or attribute questions.
• Support is purely count-based. A single large box covering many instance centres counts all of them. "No" answers are supported by grounding nothing; for comparison, that holds even when both counts are zero. Check whether this can inflate ES.
• Two models were tested differently. GeoChat and GeoGround produced boxes in a separate request that did not see their answer. The other five answered and grounded in one response, so the consistency comparison is not like-for-like.
• Recall is very low for every model (≤10.05%). Results reflect current grounding weakness as much as consistency.
• Possible confound in the No-answer exclusion. Since existence questions with a No reference answer are excluded, wrong "No" answers on images where the object is present still enter the subset.
• Small scale. This is a short v1 preprint (9 Oct 2026) with no ablations on the coverage criterion.