Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation

What it claims

Mise-en-Video (MeV) is a multi-shot text-to-video model from HKUST and KlingAI Research, posted to arXiv in October 2026. Creators place each subject as an oriented 3D bounding box per shot and per frame, and the model generates video that follows those boxes, including which way a subject faces and turns. The authors position it against MultiShotMaster, which they describe as the first controllable multi-shot framework but limited to 2D control. The backbone is a pretrained single-shot T2V model of about 1B parameters, not one of the large open models such as Wan or HunyuanVideo.

How it works

The base model has a 3D VAE, a T5 text encoder and a latent DiT trained with a rectified-flow velocity objective. Each DiT block has 2D spatial self-attention, 3D spatiotemporal self-attention, text cross-attention and an FFN. MeV adds two parts.

Subject Layout Guidance turns the 3D staging into a control signal:

1. Each subject is a time-varying oriented 3D box per frame.
2. The box is colored with a NOCS (Normalized Object Coordinate Space) encoding, so its color pattern shows orientation, such as a character turning or tilting.
3. The colored boxes are projected into each shot's camera view, giving per-frame 2D orientation maps.
4. The map sequence is encoded with the same pretrained 3D VAE into a layout latent.
5. A ControlNet-style branch, cloned from the first N/2 DiT blocks, takes this latent. Its hidden states pass through zero-initialized convolutions and are added to the base model's hidden states.

Hierarchical Context Attention splits the prompt into global, shot and subject captions and masks attention by level. Global text is visible to all tokens. Shot text is visible only to that shot's tokens. Subject text is visible only to the token regions of that subject's box. This mask is how subject descriptions are tied to specific boxes, which the authors say plain per-shot prompts fail to do when several subjects appear.

flowchart LR
  A[3D boxes per shot] --> B[NOCS color + projection]
  B --> C[VAE layout latent]
  C --> D[Control branch, N/2 cloned blocks]
  E[Global / shot / subject captions] --> F[Hierarchical attention mask]
  D -->|zero-conv add| G[Base DiT]
  F --> G
  G --> H[Multi-shot video]

The training set has 160K videos, each with hierarchical multi-shot captions, NOCS color maps and oriented 3D boxes, built with the authors' own annotation pipeline.

Evidence

The fetched text stops in the method section, before any results. It gives only the abstract's claim that MeV beats "strong baselines" on video quality, inter-shot consistency and subject-level controllability. It names no metrics, numbers or baselines.

Limits

The page has no quantitative results, ablations or stated failure modes. It does not say whether code, weights or the 160K dataset will be released, or under what licence. The 1B backbone is described only by size and components and is not named. It may be an internal KlingAI model, since the first author did this work as a KlingAI intern. Inputs at inference require per-frame oriented 3D boxes and a camera for each shot, so the page does not show how a user without a 3D staging tool would produce them. A project page is listed at heiheishuang.xyz/Mise-en-Video.