SPARK: Self-Correcting Embodied Memory via Physical Re-Observation

What it claims

SPARK, from Davidson College, the University of Maryland and Carnegie Mellon, repairs an embodied retrieval-augmented generation (RAG) memory that a robot built from its own noisy perception. It does this without ground truth and without waiting for a failed query. The best single-episode gain is 2.7× EQA accuracy on one map-condition pair. Continuing correction through Episode 4 lifts every map above its undegraded reference. The abstract says Episode 4; the contributions list says the third episode. Critic-targeted revisits beat untargeted revisits on mean gain under the same budget.

How it works

The memory is Embodied-RAG's hierarchical structure. Embodied-RAG's released AirSim code fills memory from simulator ground truth, so SPARK replaces that with a perception pipeline that can actually be wrong:

1. Segmentation. SAM2 runs at a lowered confidence threshold, to favor over-detection.
2. Labeling. A VLM sees each mask three ways: a padded crop, a masked crop, and the full frame with a box. It returns a free-text name and a category. Six background terms, such as "ground", are rejected.
3. Normalization and placement. Names are normalized into shared categories per map. Objects are placed by a high depth quantile and backprojected using the true pose.
4. Clustering. Clusters combine spatial proximity with text-embedding-3-small similarity. A co-visibility constraint requires a shared observing waypoint.

The critic uses gpt-5-mini for two signals: plausibility (is the label visually supported) and density (visible versus stored objects per waypoint). The third signal, graph coverage, is a rule that flags clusters with too few observations or nodes.

The planner merges nearby flagged clusters from one source waypoint into a single region. The robot turns in place to face the region, then steps 2.5 m and 4 m toward it if the region radius plus 2 m of clearance allows. If the region is closer than 6 m, it steps back 2.5 m instead. Steps stay on the straight line to the region, because the robot has no obstacle map. Revisits fly without the Episode 1 degradation.

flowchart TD
    A[Episode RGB-D] --> B[Perception pipeline]
    B --> C[Memory H]
    C --> D[Critic: plausibility, density, coverage]
    D --> E[Planner: revisit viewpoints]
    E --> F[Revisit observations]
    F --> G{Match within 2 m, same category?}
    G -- yes --> H[Replace stored object]
    G -- no --> I[Add new object]
    D -- flagged implausible --> J{In view, unoccluded, not detected?}
    F --> J
    J -- yes --> K[Remove object]
    H --> C
    I --> C
    K --> C
    O[Oracle graph] -. evaluation only .-> L[EQA scoring]
    C --> L

The selective merge applies three rules:

• A detection within 2 m of a stored object of the same category replaces that object.
• A detection with no match is added as an object the first pass missed.
• An object is removed only if the critic flagged it as implausible and a revisit saw its position, unoccluded in depth, without detecting its category there.

Evidence

The evaluation uses four outdoor AirSim maps under two degraded-viewpoint conditions. A fixed set of 158 EQA questions covers inventory counts, the dominant type, type richness and rare-object presence. A visibility-constrained oracle, built with field-of-view limits and raycasting, scores the answers and is never seen by the system. Ablations show each of the three critic signals contributes to performance. Selective merging matches append-only gains with a smaller memory.

Limits

• Simulation only. All results are from AirSim. Depth noise is synthetic and range-dependent.
• Ground-truth pose. Pose is exact and unperturbed, so localization error is not tested.
• Static world. Cross-episode mismatches are assumed to be errors, never scene changes.
• Simple planner. It has no obstacle map, moves only along a line, and uses fixed step lengths.
• Not shown on the page. The fetched text cuts off in the method section. Per-map numbers, thresholds and the appendices are missing, and the page mentions no code release or licence.