uclanlp/OpenVLHarness

What it claims

OpenVLHarness wraps an existing multimodal LLM in a tool-calling loop with nine capabilities and a per-question memory, without fine-tuning. On the paper's 22 benchmarks, average scores rise by 9.66 points (Kimi K3) to 12.62 points (Qwen3-VL-32B-Instruct) over the same backbone without tools. The code is MIT-licensed. The paper is arXiv 2610.26860 (2026), by Yu Zhou, Kai-Wei Chang and co-authors.

How it works

The backbone uses native tool calling. It picks tools, reads their evidence and continues until it answers or hits max_iterations. Three parts carry evidence across steps:

• Capability layer. Each tool returns a model-readable observation: annotated images, summaries or compact numerical data.
• Multimodal memory. Images, boxes, masks, depth maps and URLs stay addressable by ID for the whole question. Memory resets for each new question.
• Python coding agent. The same backbone writes code in a separate, fresh context. That code computes over stored tool results, for example the spatial relation between two grounded objects using their depth.

All tool calls go through an HTTP router. Separately from the main model, every preset uses Qwen3-VL-8B-Instruct to summarize search results and extract webpage content.

flowchart LR
  A[Agent loop + memory] --> R[Tool router]
  R --> G[GPU workers: SAM 3, PaddleOCR-VL, DA3]
  R --> Z[Zoom, local CPU]
  R --> P[Python executor on host]
  R --> S[Serper API, paid]
  S --> Q[Auxiliary Qwen3-VL-8B]
  R --> H[Public image URL: tunnel or S3/GCS]
  H --> S

Evidence

The table gives average scores over the 22 benchmarks, base model versus the harness. The benchmarks cover counting/grounding, multimodal search, general VQA/hallucination and spatial understanding.

Qwen3-VL results are means of three runs. The paper judged with Qwen3-VL-8B, and with GPT-4o on the search and deep-research benchmarks. Per-benchmark scores and comparisons with other harnesses are on the project page, not in the README.

How to use it

Install with Python 3.10 or newer:

pip install -e '.[qwen,server]'

Add the eval extra for benchmarks. Start the router with the same YAML the agent uses:

python -m openvlharness.services.server --config configs/agent/qwen3_8b_paper.yaml

Call the agent from Python:

from openvlharness.bootstrap import build_agent
agent = build_agent("configs/agent/qwen3_8b_paper.yaml")
result = agent.generate([
    {"type": "image", "value": "/path/to/image.jpg"},
    {"type": "text", "value": "How many bottles are visible?"},
])

A full deployment needs four kinds of service:

• The main model endpoint.
• A Qwen3-VL-8B endpoint for search. The 8B preset can share its server. The 32B preset needs a second server.
• Three GPU perception workers (SAM 3, PaddleOCR-VL, Depth Anything 3).
• A funded Serper key. Reverse image search also needs a publicly fetchable query image, served from a tunnelled built-in host or an S3/GCS bucket.

Capabilities can be removed from the YAML. examples/local_tools.yaml runs only zoom and coding against a hosted GPT model, with no GPU workers, Serper or cloud storage. New backends need source registration in bootstrap.py and server.py.

To evaluate, set LMUData and run openvlharness-eval. Dataset archives take about 3.6 GB and are stored twice, once in the Hugging Face cache and once extracted. Fixed 300-question subsets exist for the largest search benchmarks to cut search cost. One judge serves each invocation, so search benchmarks need a separate run with a GPT-4o judge.

Limits

• Coding tool is not sandboxed. It runs generated Python as the agent host's user, with network and filesystem access. The execution timeout is not a sandbox.
• Health checks miss broken tools. Router health checks do not detect unreachable GPU workers, missing Serper keys or image-hosting faults. These appear on first tool use. /backend-status counts silent fallbacks to raw search content when the auxiliary model times out.
• Failed rows score as wrong. Rows that failed because a service was down are scored as incorrect until rerun with --ignore-failed.
• One paper configuration is missing. The Qwen2.5-VL-7B configuration, used for the paper's training-based harness comparison, is not released.
• Fresh PixmoCount builds can differ. A new build may contain fewer than the paper's 534 images, because images that can no longer be fetched are dropped.
• Small subsets do not reproduce the paper. They check the install. Reproducing the scores needs the prompts, judges and scoring protocol in the paper's Appendix D.