How Benchmark Graph Construction Shapes Multi-Hop KG Navigation: A Controlled Audit of PathQuestion and PathQuestion-Large

What it claims

Restoring PathQuestion (PQ) and PathQuestion-Large (PQL) to their distributed source graphs changes the task sharply. Graph size, branching and answer multiplicity all grow, while the questions, answers, relation chains and splits stay fixed. For PQ, the active graph grows from 2,256 to 75,043 entities and from 3,377 to 375,444 triples. For PQL, it grows from 6,505 to 416,249 entities, from 411 to 3,076 relations, and from 5,597 to 831,940 triples. The authors also report a design problem in the released benchmarks: in 33.0% of PQL released records the topic entity is itself a valid answer, so a zero-length solution exists despite a multi-hop reference path.

How it works

The audit holds fixed the question text, topic entity, evaluation horizon, released answer set, released relation chains, structural family and split. Only the navigation graph changes, from the task-filtered graph G_F to the source graph G_S. Scoring still uses released answers and paths. Separately, it enumerates "graph-consistent" answers and paths, meaning endpoints reached by following the released chains in the active graph, as diagnostics rather than new gold.

Surface variants are grouped into structural families and split 80/10/10 at family level, so paraphrases do not leak across splits. Metrics are averaged per family first, then across families. Cyclic and backtracking reference paths are kept, and no no-repeated-entity constraint is imposed. Navigation diagnostics are reused unchanged from Hernandez et al. (2026b): the exact and Monte Carlo random-walk answer probability (RW-Ans), an answer-oracle shortest-path reference, path edit distance (PED), and edge-overlap F1_SG. With multiple references, the best match counts.

Evidence

The reconstruction keeps 7,105 PQ instances in 2,923 families and 2,114 PQL instances in 1,530 families. These come from 7,106 and 2,625 released records. Eight PQL cases carry two relation chains, and both are kept. The PQ test split has 294 families (724 instances); the PQL test split has 154 families (213 instances).

Source restoration adds new graph-consistent answers to 2,158 of 2,923 PQ families (13,307 added assignments) and to 633 of 1,530 PQL families (13,541). It adds new paths to 2,437 PQ families (50,230 realizations) and 647 PQL families (14,147).

Connectivity changes too. PQ's largest weakly connected component grows from 93.75% to 100%. PQL's grows from 38.69% to 98.33%, and its maximum out-degree rises from 115 to 26,023. Nearly all PQL 3-hop reference paths revisit an entity. Per-policy accuracy and fidelity numbers for the frozen small language models and MINERVA fall outside the fetched text.

How to use it

The datasets are not redistributed, because the original benchmarks have no explicit redistribution licence. Only the reconstruction code is released, at https://github.com/HalcyonSolutions/PathQuestion-Family. It rebuilds the benchmark versions from the original released artifacts. Install or run commands are not given in the fetched text.

Limits

The source graph G_S is a benchmark artifact, not full Freebase, and the authors avoid calling it that. Published mixed-hop results (UHop's PQ+/PQL+, SRN's PQ-M/PQL-M) use different task formulations and are not directly comparable to this reconstruction. The shortest-path reference is not an upper bound on path fidelity, since a shortcut can reach the answer off the released chain. Wording irregularities in the questions are covered in an appendix that the fetched text does not include. The fetched page is cut off before the main policy results, so the size of the accuracy drop and the partial recovery under graph-consistent scoring cannot be quantified here.