OWLS: Open-world Language-conditioned Speed Fields for Zero-Shot Rover Navigation

What it claims

OWLS turns an open-vocabulary segmenter's pixel-wise mask probability into the speed field of an eikonal minimum-time planner. Obstacle clearance and terrain appearance are therefore optimized in one trajectory, from one calibrated RGB camera plus wheel odometry. On a physical 4-wheeled rover with Grounded SAM 2 (GSAM2) and the prompt "drivable ground", it reached 100% success against static obstacles indoors and outdoors, over 5 trials each. Porting it to a new rover takes at most two radii set from the rover's dimensions. The authors are from the University of Michigan and NASA JPL, and the preprint is dated October 2026.

How it works

1. Score. The VLM returns a continuous map H in [0,1] per pixel, the mask probability before thresholding. Grounded SAM 2 and SAM 3 are used. Simulation uses the single fixed prompt "ground" for every platform, model and episode.
2. Project. A flat-ground homography maps H onto a 640×640 grid at 0.0125 m, covering 8 m ahead and ±4 m laterally. Cells outside the image or above the horizon get 0. The blind strip in front of the rover and cells hidden by vehicle parts, such as Perseverance's stowed arm, get maximum traversability instead.
3. Adapt. A grayscale dilation with radius ρ_t = 0.19 m, shared by all rovers, erases low-score blobs small enough to drive over. A Gaussian-tapered erosion with radius ρ_r then builds a decaying clearance buffer. ρ_r is 2.15 m for Perseverance and 0.50 m for Pragyan and the terrestrial rover, set from track width. Neither step thresholds the map.
4. Plan. Speed is c = s·V′ + ε, with ε = 0.1. The gain s is 8 for GSAM2 and 16 for SAM 3, to correct for their different probability ranges. The eikonal equation is solved with the Fast Iterative Method, seeded at the goal or at a boundary cell along the goal bearing. Unobserved cells move at the same floor speed as confident obstacles.
5. Track. A bang-bang controller turns in place when heading error is 8° or more, otherwise drives straight. It uses a 0.25 m lookahead and stopping radius. Odometry carries each plan forward between images.

There is no persistent map: OWLS replans from every image.

Evidence

On a mobile RTX 4090, one GSAM2 planning cycle takes 692 ms: 490 ms in the VLM and 199 ms in morphology. Planning runs at 1 Hz. Control runs at 25 Hz in simulation and 100 Hz on hardware.

Real-rover success rates (%) over 5 trials per configuration, all using a 640×480 front camera:

DINOv3-patch is a language-free control that queries with the image patch ahead of the rover. Its smoother map made it less reactive. DINOv3-lang applies the DINOv2-txt recipe to DINOv3, and it failed every trial: its text-query similarity spans about a seventh of the patch query's range, and per-frame renormalization amplifies the noise. The authors treat differences of one or two trials as indicative only.

Simulation runs in the Space Robotics Bench on IsaacLab/IsaacSim, using its Perseverance and Pragyan models on 100 procedural terrains each, with prompt and morphology ablations. The fetched text ends before any simulated results.

Limits

• The outdoor dynamic success rate was 2 of 5. The authors attribute this to under-segmentation in outdoor lighting and to phase lag between the 1 Hz planner and the faster controller.
• Ground projection assumes flat ground within the 8 m window.
• No real space rover was used. Planetary evidence is simulation plus qualitative inference on real rover imagery.
• Success means end-to-end traversal without collision. It does not measure path quality, and the method can stall at local dead-ends.
• The controller is deliberately minimal and suited only to low-speed tracking.
• The page gives no code, weights or licence.