Anionex/agent-vision-toolkit

What it claims

The repo argues that a text-only coding agent's vision can live in the harness rather than in the model. Its main idea is intent-aware captioning. Many vision bridges turn an image into a generic description. This toolkit instead takes the "viewing intent" and sends it to the vision model as a focus hint. The intent comes either from the user's message or from the assistant's stated reason for calling its built-in image tool (e.g. Codex's view_image). The author claims this gives task-relevant detail at lower cost, higher accuracy and faster responses. The page gives no benchmark for that claim; the evidence is demos run with DeepSeek V4 in Codex. Those demos are:

• chess played via ground
• UI restoration from a hand-drawn sketch (deepseek-v4-flash)
• infographic-to-HTML reconstruction
• a "fast mode" page rebuild aimed at a first screenshot in about three minutes

How it works

flowchart LR
  A[Agent e.g. Codex] --> P[Proxy 127.0.0.1:19100]
  P -->|no image| U[Text-only upstream]
  P -->|image present| H[Extract intent: user msg or tool-call reason]
  H --> V[Vision model]
  V -->|text description replaces image| U

• Proxy auth: it forwards the agent's existing Authorization header unchanged. Don't re-store the upstream key; the proxy needs only VISION_*.
• Plugins instead of proxy: Pi, Oh My Pi and OpenCode use single-file native extensions rather than the proxy. These have their own settings and call /chat/completions only.
• trace is not an LLM: it is deterministic and local. It finds the centerline and fits SVG <circle>/<line>/<polyline>/<polygon> primitives in source coordinates. --outline falls back to a filled silhouette via vtracer.
• ground and detect return boxes in original-image pixel coordinates. --region X1,Y1,X2,Y2 zooms into small targets while still reporting original coordinates. Full-screen detect is described as only a first draft; on dense screens, inventory region by region.
• Long-screenshot OCR finds safe cut bands and OCRs each chunk with glance. It then merges the overlaps and writes a boundary audit.
• Proxy egress goes direct by default and ignores the Windows system proxy. --upstream-proxy / VISION_UPSTREAM_PROXY sends traffic through a CONNECT tunnel, and --proxy-first tries that proxy before direct. Routes switch only on connection failures (refused, DNS, TLS, or a 5 s timeout). HTTP errors pass through unchanged. If every route fails, the proxy returns a 502.

How to use it

Put the config in ~/.config/agent-vision-toolkit/env with chmod 600:

VISION_API_KEY=sk-...
VISION_BASE_URL=https://openrouter.ai/api/v1
VISION_MODEL=google/gemini-3.6-flash

Protocol and language options:

• Use VISION_API_PROTOCOL=responses for OpenAI Responses.
• Use VISION_API_PROTOCOL=anthropic for Anthropic Messages, with a base URL ending in /v1.
• Descriptions default to Chinese. Set LANG=en for English.

git clone https://github.com/Anionex/agent-vision-toolkit.git
export PATH="$PWD/agent-vision-toolkit/bin:$PATH"
npx skills add Anionex/agent-vision-toolkit --skill vision-skills -a codex -g --copy -y

Dependencies:

• glance needs only Python 3.11+.
• ground, detect, crop and the OCR script need pillow.
• trace needs pillow and numpy.

Example calls:

glance screenshot.png -q "What is the dominant color of this image?"
ground screenshot.png "Send button"
detect page.png --region 238,600,953,671
trace screenshot.png --region 1563,514,1668,621 -o icon.svg
python3 skills/vision-skills/scripts/long_screenshot_ocr.py long-chat.png --mode chat -o long-chat.ocr.md

Recent changes:

• 2026-08-18: the skill was renamed vision-skills (was vision-tools).
• 2026-08-13: a DeepSeek Harness package (dsh-vision-toolkit, 10 tools) was added as a git submodule. Clone with --recurse-submodules.

For cheaper vision calls, the author suggests a local side model such as Gemma 4 or Qwen 3.5/3.6.

Limits

• This is image-to-text only. No vision tokens reach the text model, and quality depends on both models together.
• The proxy cache lives in the process only and is cleared on restart.
• Proxy authentication for an upstream proxy is not supported.
• The native plugins support /chat/completions only.
• Every quality claim rests on demos, with no quantitative comparison.
• The page states no licence.