Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

What it claims

Code2Video-Compass (arXiv, 20 October 2026, Fudan University and Sun Yat-sen University) tests whether multimodal LLMs can write a full web-animation project that renders into a complete video, typically over one minute long. Tasks are 718 T2V, 693 I2V and 693 V2V. The authors report two headline findings. First, scores drop as task complexity rises. Second, video quality and execution reliability "do not always align": a model that runs reliably does not necessarily produce the best video, and the reverse also holds.

How it works

Models write code for a fixed stack: HTML/CSS, SVG, Canvas and Three.js, rendered with Remotion. In I2V and V2V, the model must decide where the reference belongs in the timeline. It must then rebuild the reference in code; embedding or replaying the reference image or clip is not allowed.

Each task is labelled on three axes, each with three levels:

• Construction: primitives, then custom geometry, then procedural or simulation systems.
• Dynamic animation: staged sequences, then coordinated changes across scenes, then concurrent transformations.
• State dependency: consistency across adjacent stages, then intervening stages, then long-range dependencies.

Tasks come from two sources that meet at a shared annotation step:

flowchart LR
  A[VidProM + Shot2Story texts] --> I[Independent set: 385 tasks]
  B[LICA, SciDraw-6K, SynthChartNet, PD12M images] --> I
  C[YouTube clips] --> I
  D[YouTube, Bilibili, REDnote, X videos] --> P[Paired set: 573 videos x 3]
  I --> Q[Prompts, QA checklists, complexity labels]
  P --> Q

The paired set starts from 197 YouTube records and 2,473 candidate URLs. Gemini 3.8 Flash screening keeps 979 videos. Deduplication with SSCD frame embeddings indexed in FAISS leaves 915. Human review keeps 573, and each one yields aligned T2V, I2V and V2V tasks (1,719 in total). V2V references are 2 to 10 second clips.

Scoring has two levels:

• 12 objective metrics: execution success, token consumption, technical and aesthetic quality, motion smoothness, local and global text-video alignment, and image and clip fidelity.
• 5 MLLM-judge metrics: a checklist QA for instruction following (6 dimensions, up to 16 questions per task) and four holistic scores for content coherence, visual design, motion quality and viewer engagement.

Evidence

The paper's Table 1 compares this benchmark with prior code-based benchmarks:

Prompt lengths use the Qwen3 tokenizer.

The 15 evaluated models are:

• GPT-6.1 Sol, GPT-6 Astra and GPT-6 Luna
• Claude 5.5 Opus, Sonnet and Haiku
• Qwen 3.8 Max, Qwen 3.8 Flash and Qwen 27B
• Gemini 3.8 Flash
• Kimi K3
• DeepSeek V4.1 Flash
• GLM 5.3 Flash
• MiMo V2.6 Pro
• MiniMax M3

How to use it

The project page is https://code2video-compass.github.io/. The fetched text gives no install commands, data licence or harness details.

Limits

The fetched text ends at the dataset statistics, so no per-model scores or results tables are available here. The model used as judge is not named in the captured text. LLMs shaped the data:

• Gemini 3.8 Flash screened and scored the source videos, picked keyframes and clips, and drafted the prompts.
• GPT-6.1 Sol helped curate text seeds and refine the checklists.

Human review was done by three co-authors. Source videos come from YouTube, Bilibili, REDnote and X, and the captured text does not address their redistribution terms. Results apply only to the HTML/CSS, SVG, Canvas and Three.js stack rendered with Remotion. Other code-video tools such as Manim are not tested.