Onboard Lightweight LLM Reasoning for Human-Aware Safe UAV Navigation

What it claims

A semantic layer for UAV planners can run fully onboard a Jetson Orin NX (16 GB, 25 W). It turns RGB-D frames into per-category clearance boxes that an unmodified planner consumes. The LLM (Qwen-2.5-0.5B-Instruct, 4-bit) never commands the vehicle. It only outputs {is_dynamic, bbox_size} per object class. A persistence layer keeps pedestrians in the map through occlusions. The authors show the layer is planner-agnostic by plugging it into two back ends: EGO-Planner for urban navigation, and kinodynamic A* plus nonlinear MPC for landing.

How it works

Perception. A frozen BLIP ViT-B/16 encoder (about 86M parameters) is shared by two heads:

• BLIP's own captioning decoder, which produces a scene caption of about 30 tokens.
• A trained, text-prompted segmentation decoder. It fuses multi-scale features, conditions on the prompt via FiLM, and uses four bilinear-upsample/3×3-conv blocks.

Each mask component is back-projected through depth to a 3D centroid, after rejecting depths more than 2σ from the in-mask median.

Reasoning. Reasoning happens once per class, not once per frame.

flowchart TD
  D[Detection: class + centroid] --> C{Class in verdict cache?}
  C -- hit --> V[Cached verdict, no inference]
  C -- miss --> R[Hybrid RAG: bge-small + BM25, RRF]
  R --> L[Qwen-2.5-0.5B, GBNF JSON grammar]
  L --> G{Valid JSON?}
  G -- yes --> S[Store in cache]
  G -- no --> F[Conservative default box]
  S --> V
  V --> P[Unsafe box into planner map]
  F --> P

• Cache warm-start. The cache is pre-filled by running the same RAG→LLM path for the prompted classes: person, car, truck.
• Retrieval. bge-small-en-v1.5 (under 100 MB) handles synonyms, and BM25 anchors keywords. Results are fused with Reciprocal Rank Fusion (η=60), and only the top-k results go into the prompt.
• Inference. The model runs on llama.cpp with importance-matrix Q4_K_M quantization and a statically allocated KV cache.

Action.

• Tracking. Greedy nearest-neighbour association within each class (2 m gate, 4 s time-to-live) gives persistent IDs.
• Occlusion hold. An object's last box is held in place for T_h = 3 s before its voxels are released. Pedestrians are deliberately not extrapolated.
• Why the hold was added. The planner's default 2 s map decay dropped occluded pedestrians before they reappeared.

Evidence

Hardware choice.

• The Orin Nano (8 GB) could not fit the full stack.
• The Intel NUC 13 Pro draws over 40 W, weighs 256 g decased, and lacks tensor cores.

Cloud rejected. About 100 kbps uplink means roughly 8 s to send one 384×384 JPEG. Hosted models add 1.6–2.8 s of JSON-generation latency.

Modular vs. monolithic VLMs on Orin NX (Table I):

LLM selection.

• Gemma-2, Phi-3.5, and Llama-3.2-1B at IQ4/Q3 often broke the output schema.
• Llama-3.1-8B fits in memory but takes more than 3.3 s per query.
• Qwen-2.5-0.5B answers in 0.75–0.79 s with 5.6 GB total system memory.

Urban navigation (Gazebo, 20 runs per arm, about 106 s route, 2 SUVs and 2 walkers):

• Collision events per run fell from 1.35 to 0.45 (p=0.038). This is the only significant safety metric.
• Worst-case clearance rose from 0.16 to 0.52 m, which is not significant.
• Crashes were 3 vs. 1 out of 20, which is within noise.
• The costs were longer paths (54.5→68.7 m) and 2.6× peak jerk.
• Perception held 9.96 Hz.

Landing (50 trials per cell; success means landing within 0.5 m without entering the actor's 1 m buffer):

The unaugmented LLM is worse than no LLM at all, because malformed or hallucinated verdicts destabilize the planner. RAG plus grammar-constrained decoding is what makes the 0.5B model usable.

Limits

• Simulation only. All closed-loop results are in Gazebo (ROS Noetic, simulated RGB-D). The simulator is non-deterministic, with 0.5–0.8 m per-run clearance standard deviation, so differences under about 0.2 m cannot be resolved at n=20.
• Online reasoning untested in navigation. In the urban benchmark every label hit the warm cache, so no online LLM query was ever issued. The cache-miss path, the claimed route beyond a fixed semantic policy, is not exercised there. Only the landing ablation tests the reasoning path.
• Short range for small objects. The 224-pixel encoder input limits detection of small or distant objects, especially pedestrians. This is mitigated only by larger pedestrian margins and persistence.
• Noisy tracking. On average 23% of the 45 IDs created per run die after a single frame (background artifacts).
• No release details. The fetched text mentions no code, weights, licence, or knowledge-base contents, and it cuts off mid-conclusion.