ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

What it claims

ViTeX-Bench is a benchmark for replacing scene text in real video (signs, whiteboards, labels, jerseys) while keeping the rest of the clip unchanged. It is accepted at the NeurIPS 2026 Evaluations and Datasets track.

Each task is a tuple: source video, per-frame text-region mask, and a source→target string pair. Every clip is fixed at 120 frames, 1280×720, 24 fps, drawn from Panda-70M and InternVid. The frozen 157-video evaluation split covers four scripts: Latin, Chinese, Japanese and Cyrillic. Source strings run 1–41 characters and target strings 1–36. Source and target are roughly length-matched within each pair. About 25% of clips are static and 75% have dynamic text motion. Text is small: the mask covers 0.032±0.022 of the frame on average.

How it works

Training pairs come from a human-in-the-loop pipeline that builds four assets per video:

• Mask. An annotator places keypoints on the first frame. SAM 3 propagates the mask to the remaining frames, and the mask is then dilated.
• String pair. Qwen3-VL-32B-Instruct reads the source string and proposes a similar-length target. An annotator audits it.
• Clean background. removal-1.3B, a fine-tuned Wan2.1-VACE-1.3B from PISCO, erases the glyphs together with their shadows and highlights.
• First-frame patch. Gemini 3 Pro Image (Nano Banana Pro) rewrites the first frame. A second SAM 3 pass cuts out the new text.

flowchart LR
  A[Clean background] --> C{Text static?}
  B[First-frame target patch] --> C
  C -->|static| D[Strategy A: alpha composite]
  C -->|static| E[Strategy B: fine-tuned PISCO]
  C -->|dynamic| E
  D --> F[Keep better output]
  E --> F

Strategy A only works when the text never moves. Static clips run both strategies, and the better result is kept. The final split has 56 Strategy-A and 174 Strategy-B videos. These pairs are training references, not unique ground truth.

Text correctness scoring. PP-OCRv5 reads the dilated-mask crop on each frame. Scoring is restricted to "source-detectable" frames, where OCR of the source frame reaches at least 0.5 similarity to the source string. Strings are compared with a substring edit distance, so extra OCR characters around a correct target are not penalised. From this the benchmark computes three scores:

• SeqAcc: exact match.
• CharAcc: partial credit for near-correct renderings.
• TTS: whether the decoded string stays the same across adjacent frames. It measures stability, not correctness.

The other axes use Flicker, Warp and MUSIQ (frame/crop) for quality, and PSNR, SSIM, LPIPS and DreamSim for locality.

ViTeX-Edit-14B fine-tunes Wan2.1-VACE-14B on the 230 training pairs. It adds a glyph-video stream that renders the target characters along the trajectory of the source text.

Evidence

The paper groups baselines into four editing families, each with a characteristic failure:

ViTeX-Edit-14B reaches CharAcc 0.688, the best mean among video-native editors. It also has the lowest text-crop Warp among raw outputs. The fetched text does not include the full results tables.

Annotation reliability. An independent annotator redid the masks on 12 difficulty-stratified clips. Agreement was mask IoU 0.95 and Dice 0.98. Rankings stayed stable under the alternative masks: Kendall τ was 0.94 for DreamSim-loc and 1.00 for text-crop Warp.

How to use it

The project page is https://vitex-bench.github.io/. To submit results, produce 120-frame 720p edits for the 157 frozen clips. Edited references for those clips are withheld. The fetched text gives no download commands, weight locations or licence. Check the project page for these.

Limits

• Single construction annotator. One author did the original annotation. The reliability audit covers only 12 clips.
• OCR-dependent scores. Correctness depends on PP-OCRv5. Frames where the source text is unreadable are excluded, so heavily blurred or occluded text is under-scored.
• Synthetic training targets. The edits come from generative models: Nano Banana Pro plus PISCO or compositing.
• Fixed clip format. All clips are 5-second, 120-frame videos. Longer clips are not covered.
• Truncated source. The page cuts off in the metric section, so the per-axis primary metrics, full results, human-evaluation findings and the stated limitations are not available here.