Explaining Navigation Decisions: Stage-Aware Visual Attribution for Vision-Language Navigation

What it claims

The paper is an attribution method for VLM-based VLN policies that explains which visual evidence, in current and past frames, supports a chosen action. It ties that evidence to the part of the instruction it serves. The authors name the failure it targets "instruction-stage insensitivity": existing explainers such as DEX-AR give nearly identical token and visual attributions before and after the agent moves to a new sub-instruction. On R2R and RxR (VLN-CE), the method beats DEX-AR by 34.58% and 35.37% in Precision and by 25.72% and 23.50% in F1. The authors describe decision faithfulness only as "competitive", not better.

How it works

The method is gradient-weighted attention, applied per frame and kept at instruction-token resolution. It needs white-box access to the policy.

1. Temporal attribution. For each observation, it takes the instruction-to-patch attention submatrix from layers 8 to 15, all heads. Each entry is multiplied by the gradient of the selected action's logit, and only positive values are kept. Heads are weighted by maximum absolute gradient. This needs a fully differentiable forward pass through multimodal prefill.
2. Stage estimation. Each frame's map is summed over patches to give a distribution over instruction tokens. The expected normalized token position, µ, is a scalar "stage" for that frame.
3. Stage-guided temporal refinement. Historical frames are weighted by a softmax over −|µ_current − µ_t| / τ. Their token distributions are averaged. The current frame's map is reweighted per token by δ + (1−δ)·that average, so δ sets a floor that protects newly seen evidence.
4. Instruction-guided spatial filter. This step adapts a rank-Gaussian filter from TAM. Each patch gets a preservation score: its strongest instruction token, using the geometric mean of patch-to-token and token-to-patch concentration. Patches with weak associations are suppressed more. Compact, strongly grounded regions survive.

ISS, the closest embodied-action baseline, masks the input and reruns the policy many times. This method needs no repeated policy evaluations.

Evidence

• Policies: NaVILA (384×384 input), NaVid (224×224) and StereoNav (448×448).
• Setting: VLN-CE in Habitat. Grounding is scored against the simulator's semantic scene annotations.
• Baselines: All baselines explain the same selected action on identical inputs and observations.
• Hyperparameters: τ_H = 0.25, δ = 0.7, k = 3.
• Faithfulness metric: It is a new input-perturbation metric, "Action-Space Delta-NLL".
• Other demonstrations: The authors show failure-case diagnosis and a real-robot run on a Unitree G1.

How to use it

The page gives no code, repository, licence or runtime cost. Reproducing it requires:

• attention matrices and gradients from the policy's language model;
• knowledge of which sequence positions hold instruction tokens and which hold each frame's patches;
• the history frames the policy itself selected.

Limits

• The fetched text stops at the metrics definition. The per-policy result tables, faithfulness numbers, ablations of temporal refinement and the spatial filter, and the G1 and failure-diagnosis details were not available. The headline gains above come from the introduction.
• Faithfulness is reported only as "competitive" with existing methods. Better localization against semantic labels does not show that the model actually relied on those regions.
• µ is a token-position average. It assumes instruction stages run in order through the text, which non-linear instructions may break.
• Quantitative evaluation is simulation-only, on R2R and RxR. The real-robot result is a demonstration of applicability.
• Layer choice (8 to 15) and hyperparameters are given for the tested policies. The page does not say whether they transfer to other architectures.