Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving

What it claims

GeoCoTDrive has a driving VLA do three things in order: name the image regions that matter for the next manoeuvre, pull 3D features only from those regions, then decode the trajectory. The paper also releases PlanningGrounding, a VQA-style dataset of 146K planning-relevant region annotations. On nuScenes open-loop planning, it reports 0.11% average collision rate at an average L2 of 0.32 m. The best prior VLA in the table is LaST-VLA at 0.18%. Code is at https://github.com/TabGuigui/GeoCoTDrive.

How it works

Inputs are a single front-view image, a driving command (left, right, straight, lanefollow), and ego status (velocity, acceleration, position).

flowchart LR
  I[Front image + command + ego] --> V[VLM]
  V --> B[2D boxes: grounding text]
  I --> G[Frozen geometry model]
  G --> F[Dense feature map]
  B --> S[Grid sampler]
  F --> S
  S --> M[MLP align head]
  M --> V2[Geometry tokens interleaved]
  V2 --> T[Trajectory tokens]

The steps:

• Grounding. The VLM first autoregressively emits up to N boxes (xmin, ymin, xmax, ymax). Each box has a category and a reason.
• Categories include critical-object, road-boundary, occluded-unknown-area, conflict-area and dense-object area.
• Many of these are non-object regions, such as curbs, blind spots and conflict zones, which object grounding misses.
• Geometry retrieval. A frozen geometric foundation model encodes the same image. The paper instantiates two: Depth Anything V3 and VGGT, using intermediate backbone features.
• A regular grid of K points inside each box is bilinearly sampled.
• An MLP projects the samples to the VLM hidden size, and they are inserted after the grounding text.
• Training. Plain autoregressive LM loss under teacher forcing, supervised on both the grounding tokens and the trajectory tokens.
• Prompt, image-placeholder and geometry tokens are masked out of the loss.
• The geometry model is not trained; only the alignment head and the VLM are.
• Dataset construction. Semi-automatic. Gemini-3.1 is prompted with the image, ego state, command and a category guide to propose boxes with reasons. A parsing and validation step follows, including a human check of box validity and tightness.

The paper positions this against two alternatives. One injects 3D-perception agent and map tokens, which needs human 3D annotation. The other injects global geometry tokens; VGGDrive, SpaceDrive and LaST-VLA are the examples. The difference here is that geometry is local and selected by the model's own grounding.

Evidence

The table uses nuScenes val with the OmniDrive evaluation protocol. GeoCoTDrive's row uses ego status in the planner but not in the BEV, the same setting as VGGDrive, LaST-VLA and SpaceDrive.

What the table shows:

• Collision is where GeoCoTDrive clearly wins.
• L2 is on par with the other VLAs, not better.
• Intersection, the road-boundary violation rate, is worse than SpaceDrive's at every horizon reported: 4.25% vs 2.79% at 3s.

Limits

• Truncated page. The fetched text cuts off partway through the dataset section.
• The abstract claims results on "multiple" benchmarks, but only the nuScenes table is visible.
• No ablations are shown, so there is no evidence here on which geometry model was used for the table, or on the effect of K or N.
• The VLM backbone, compute and licence are not stated in the visible text.
• Open-loop only. The visible results are nuScenes open-loop numbers. Closed-loop behaviour is not shown.
• Single camera. The model is front-view only.
• Synthetic labels. The grounding labels are Gemini-generated with a human check, so their quality depends on that pipeline.
• Inference cost. Inference must decode the grounding text before the trajectory, and every step needs a geometry-model forward pass. The paper reports no latency.
• Boundary adherence. The weaker intersection rate suggests that local geometry has not improved boundary adherence relative to SpaceDrive's global positional encoding.