IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

What it claims

IFC-MCQ generates multiple-choice benchmarks directly from any parseable IFC file and grades LLM answers by exact match on the selected option set. It comes from Rabindra Lamsal, Sisi Zlatanova and Xuesong Shen (UNSW Sydney) and was posted to arXiv on 21 October 2026. The authors frame it as filling a gap in existing IFC benchmarks. IFC-Bench v2 (1,027 questions, 37 models) uses free-text answers graded by an LLM. IFC-QA (99 questions) automatically grades the generated query rather than the answer the user receives. Most IFC querying systems were tested on 28 to 135 hand-written questions with manual grading.

How it works

Each item is a tuple of question, options and correct subset. A response counts only if the chosen set exactly equals the correct set. Items come in three types: single-choice, boolean (Yes/No or True/False) and multi-choice (select all correct).

The taxonomy is split by the IFC mechanism a question must read, so failures can be traced to a specific retrieval type:

• Direct retrieval: class, attribute, property, boolean property, existence.
• Relational: type object, spatial structure, material, openings and fillings, space boundaries.
• Quantitative: counting, quantity, aggregation, comparison/extremum.
• Geometric: geometry only.

Subtypes vary the direction of traversal, for example opening to host versus host to fillers.

Distractors are built to mimic IFC-specific retrieval errors: values from other entities of the same type, quantities in the wrong unit, and plausible values for properties the queried entity lacks. They are drawn from the file where possible and from fixed category rules otherwise.

flowchart LR
  F[IFC file + seed] --> G[Category generator]
  G --> K[Stem, key, distractors]
  F --> V[Independent re-derivation]
  K --> C{Keys agree?}
  V --> C
  C -->|yes| B[Benchmark item]
  C -->|no| D[Discard]

The re-derivation path never sees the stored answer key, and any candidate where the two answers disagree is dropped. Output is deterministic given the input file, the implementation version and a fixed random seed.

Evidence

The setup is as follows; the fetched text ends before any result numbers:

• Coverage study: generation and verification run on ten IFC models of varying size and modelling style.
• Stage 1: 17 proprietary and open-weight LLMs evaluated with a tool agent on two common reference IFC models.
• Stages 2 and 3: a smaller set of LLMs across those two models plus four more, adding a code agent, to separate the effects of source model and agent design.

The abstract reports category-specific failure modes and performance that varies with source model and retrieval interface.

Limits

• Scores are not comparable across models. Benchmark size depends on what each file contains and on a per-category generation cap, so categories a file cannot support are absent.
• Geometry is limited to axis-aligned bounding boxes. This covers extent comparison, base elevation and point-in-box queries only. Boolean intersection, point-in-solid and volume computation are left for future work.
• The taxonomy is stated as non-exhaustive.
• Text is truncated. The fetched page stops in Table 3, before the results, so per-category accuracies, model rankings, code release and licence are not available here.