inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

What it claims

inkGround tests whether general-purpose VLMs can trace the pen path of one named word on a full, camera-captured page of cursive. Of seven VLMs given one shared in-context prompt, Gemini-3.1-Pro scores best on trajectory fidelity and readability. The four open-source models have near-zero readability, with Accurate Rate between 0.034 and 0.075. A specialist offline-to-online model, InkSight, still beats every VLM on the recorded IAM-OnDB trajectories.

How it works

The 537 pages are cursive personal notes and greeting-card messages from a greeting-card company, whose name is withheld for review. The source has no writer labels, so each page is treated as one writer. A page averages 34.89 words (range 2 to 140), 304.3 strokes and 49 points per stroke. A word box is small: its median is 0.100 of page width and 0.037 of page height, so the model must find the word before tracing it.

Reference trajectories are pseudo-labels, built in two stages:

flowchart TD
  A[2,984 candidate pages] --> B[InkSight Small-p derenders each word]
  B --> C{DeepSeek-OCR: render vs original, >=85% words match?}
  C -- no --> X[discarded]
  C -- yes, 777 pages --> D{All raters score 10/10?}
  D -- no --> X
  D -- yes --> E[537 benchmark pages]

InkSight Small-p is prompted with "Recognize and derender." DeepSeek-OCR reads both the original page and the rendered strokes, and only 26% of candidates pass the 0.85 threshold. Then 85 raters, aged 18 to 50, score each surviving pair from 0 to 10. Every pair gets at least two raters, and a pair is kept only if all of them give 10. Raters agree on accept or reject for 81.7% of pairs.

Each model gets the page, the target word and one example image with its strokes. The prompt format follows SketchAgent's numbered-grid drawing. DTW and Chamfer distance are computed per word on coordinates normalized by word height. AR and CR come from DeepSeek-OCR reading the rendered strokes. Style is the true-writer probability from a ResNet-18 writer classifier. On inkGround that classifier treats each page as a writer and trains on 75% of each page's words.

Evidence

inkGround, against pseudo-label references (DTW / Chamfer / AR / Style):

• Gemini-3.1-Pro: 0.105 / 0.052 / 0.812 / 0.861
• Claude-Opus-4.8: 0.209 / 0.119 / 0.768 / 0.283
• Kimi-K3: 0.171 / 0.091 / 0.581 / 0.927
• Best open-source: Qwen3-VL-32B has the best Chamfer at 0.222, but its AR is 0.053.

On IAM-OnDB, against recorded ground truth, InkSight scores 0.062 / 0.042 / 0.908 / 0.967. Gemini-3.1-Pro scores 0.088 / 0.047 / 0.864 / 0.889. Open-source DTW ranges from 1.518 (Qwen3-VL-32B) to 7.743 (Molmo-7B-D). The authors note a trade-off among the closed models. Kimi-K3 has the highest style score on both datasets but the lowest readability. Claude-Opus-4.8 is readable but scores 0.283 on style on inkGround.

Limits

• References are pseudo-labels. A reference is one plausible stroke order. It is not the writer's recorded order or direction.
• InkSight on both sides. The inkGround references are InkSight outputs, and InkSight is also the specialist baseline. InkSight's IAM-OnDB result is in-domain, since it was trained on that dataset's training set.
• One OCR does two jobs. DeepSeek-OCR filters the references and also scores AR and CR.
• Style scores do not compare across datasets. The classifiers choose among 537 pages and 217 writers, so chance levels differ.
• The fetched text is truncated. It cuts off at Sec. 5.2, so the text shown does not explain how invalid or incomplete outputs are scored. It also gives no IAM-OnDB test-set size and no download location.
• Licence. The copyright holder agreed to release the data for research purposes only. No licence is named.