VideoGen-Agent: Reinforcing Video Generation Agents

What it claims

VideoGen-Agent is a multimodal agent that learns to orchestrate external tools (augmentation, generation, verification) over multi-turn interactions rather than generating video directly. Beyond what the summary states, the page specifies two things the reader can act on: the reward decomposes into three distinct signals, and there is a human-preference result the summary omits — raters prefer the tool-upgraded agent over the strongest standalone baseline in 84.3% of comparisons.

How it works

A single shared policy is trained on a category-balanced dataset spanning six tasks. Training is two-stage:

1. Supervised fine-tuning on teacher-generated trajectories to establish tool-use behavior.
2. Reinforcement learning with a category-aware hybrid reward that scores three things separately: tool-call validity, task-appropriate tool selection, and generated video quality.

At inference the agent reads the prompt and intermediate observations (verification outputs) to decide the next tool call. The key structural claim: because the policy learns tool use rather than pixel generation, swapping in a better generation tool raises the score with no additional agent training.

flowchart TD
    P[Prompt] --> A[Agent policy]
    A -->|augment| AU[Augmentation tool]
    A -->|generate| G[Generation tool]
    A -->|verify| V[Verification tool]
    V -->|observation| A
    A --> OUT[Video]

Evidence

On VABench (600 held-out prompts across procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, multi-shot temporal structure):

• Base text-to-video generator: 56.5
• VideoGen-Agent: 75.6 (+19.1)
• With upgraded generation tools, no retraining: 86.1
• Human raters prefer the upgraded config over the strongest standalone baseline in 84.3% of pairwise comparisons.

The single- vs multi-entity identity split and the six-task training breakdown are named but not scored per category on this page.

Limits

This is an abstract-only page. It gives no per-category scores, no base or upgraded tool names, no model size, no training compute, no dataset size beyond "category-balanced," and no code, weights, or licence. The VABench metric (a 0–100 score) is unnamed and its composition is not defined here. The transfer claim ("benefits from subsequent advances in generation tools") rests on a single tool-upgrade experiment; the ceiling and which tool was swapped are unstated. Check the PDF for the reward formulation and category breakdowns before building on it.