A Multimodal Engineering GraphRAG Framework for Post-Wind-Event Building Assessment and Recovery Inference

What it claims

The framework is an engineering GraphRAG system from École de Technologie Supérieure and Polytechnique Montréal. It turns post-tornado photos and reconnaissance text into building safety, habitability, functionality, recovery actions and downtime estimates. Its central argument is that semantic similarity is not engineering compatibility. Roof-covering loss, roof-structure failure, a tarped roof and an explicitly intact roof read alike to a text retriever but imply different recovery. Retrieval is therefore driven by graph structure and compatibility rules, and embedding similarity is used only as a secondary tie-breaker.

How it works

Each building is stored as a scenario with ten fields: source metadata, building information, component-level damage records, physical consequences, safety, habitability, functionality, recovery class, recovery actions and downtime. Each damage record holds component, damage mechanism, severity, evidence certainty, supporting evidence and source. Knowledge-base scenarios come from reconnaissance reports and publications, including the StEER Network, and from images.

A controlled vocabulary maps raw terms to canonical components, mechanisms and severities, and keeps the original wording and provenance. Component and mechanism "families" allow broader matches, but the authors state that family membership does not make concepts interchangeable. The graph is directed and typed. It links evidence to canonical concepts to consequences (for example, envelope breach) to recovery actions.

For a new case, a VLM writes a neutral description of only what is directly visible. Multiple images are merged only after they are confirmed to show the same building. Observations are sorted into four classes:

• confirmed damage
• explicit negative evidence (seen intact)
• unresolved (obscured or indeterminate)
• temporary measure (a tarp or boarding, with the hidden condition left unassigned)

Anything not mentioned stays unasserted rather than undamaged. No consequences or recovery states are assigned at this stage. After validation, the structured query drives graph-aligned retrieval. Compatibility, contradiction, evidence-precedence and abstention checks gate that retrieval. Recovery actions follow from supported damage–consequence links. Downtime uncertainty is propagated through an action-dependency network by Monte Carlo simulation.

A second mode answers questions about concepts and stored scenarios. A chat assistant keeps the current case context so follow-up questions do not rerun the pipeline.

flowchart LR
  I[Images] --> V[VLM: observable only]
  T[Text report] --> S
  V --> S[Structured observations, 4 evidence classes]
  S --> G{Graph retrieval + checks}
  G -->|unsupported or contradictory| A[Abstain]
  G -->|supported| C[Consequences and condition states]
  C --> R[Recovery actions]
  R --> M[Monte Carlo downtime]

Evidence

The evaluation uses held-out building scenarios. GraphRAG is compared against BaselineRAG and Structured Baseline RAG, with ablations removing engineering context, evidence safeguards and graph-path traversal. The fetched text reports only that GraphRAG "improves agreement" on key outcomes. No metrics, dataset sizes or numbers are included.

Limits

The fetched text stops partway through the methodology (section 2.3.1). Results tables, dataset size, the specific VLM and LLM used, compute, and any code or data release are not visible. The demonstration covers only tornado-affected low-rise residential buildings. Hidden or concealed damage is outside what the image stage asserts. Reference labels for safety, recovery and downtime partly depend on "engineering interpretation" rather than observed outcomes. That affects what "agreement" measures.