OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

What it claims

OpenVLHarness is a tool-use harness for multimodal LLMs from UCLA and Apple, with code at github.com/uclanlp/OpenVLHarness. The authors argue that extra reasoning budget alone yields little on evidence-heavy visual tasks. On their 22-benchmark study, moving GPT-6 Luna from xhigh to max effort multiplies model-token spend by 2.7 but raises the average score by only 0.7 points. Adding tools with persistent artifact memory gives larger gains. The authors report that the two effects are complementary, since the harness also improves with higher reasoning effort.

How it works

The MLLM acts as orchestrator. It calls task-level tools (zoom-in, grounding, OCR, depth, camera pose, text search, image search, and a coding agent). Each tool is defined independently of its backend, so a backend can be swapped without changing the orchestrator's tool vocabulary. The named backends are Segment Anything, Depth Anything, PaddleOCR-VL, search engines and image processors.

Evidence rendering turns raw outputs into an annotated image, a short description and compact structured data. For example, grounding returns an overlay plus bounding boxes on a 0-1000 scale. Every artifact is stored in per-query memory under a handle such as tool_image_6 or segment_cabinet (a dict with bbox and RLE mask). Later calls can then reuse the underlying numbers, not just the rendered picture.

The coding agent receives a natural-language objective and writes Python over the stored data. It runs on the same backbone model as the orchestrator. In the worked example, the system answers "How far apart are the window and cabinet centers?" by combining segmentation masks, a depth map and camera pose, and gives 1.1957 m.

flowchart LR
  Q[User query + images] --> O[MLLM orchestrator]
  O -->|tool call| C[Capability layer]
  C --> B[Backends: SAM, Depth Anything, OCR, search]
  B --> R[Rendered evidence]
  R --> O
  B --> M[(Persistent memory: images, dicts, URLs)]
  M -->|handles| K[Coding agent]
  O -->|objective| K
  K --> O
  O --> A[Final answer]

Evidence

Average score over 22 benchmarks, base model versus base plus harness (Fig. 1a):

The benchmarks span four groups:

• Grounding: FSC-147, CountQA, OdinW-13.
• Visual search and deep research: MMSearch, FVQA, BC-VL, RealXBench.
• General VQA and hallucination: SimpleVQA, FREAK.
• Spatial reasoning: MMSI-Bench, ERQA, MindCube.

GPT-6 Luna at medium effort with the harness beats Luna alone, Luna with Codex, and Luna with Visual Sketchpad, each at max effort, by 7.7 to 8.5 points. It does so at no more than 37% of their model-token cost, averaged over 1,000 query rollouts.

How to use it

The page names two adaptation workflows that need no weight updates. "Harness search" uses a coding agent to edit prompts, tool descriptions and output templates. "Tool generation" composes existing capabilities into new tools. The fetched text gives no install commands, licence or hardware requirements.

Limits

The fetched text breaks off after the introduction. Ablation results for the capability layer and memory are described but not shown. The cost comparison counts model tokens only. It is unclear whether it includes compute for the perception backends or search calls. Memory persists within a single query, not across queries.