Connected Self Forcing: Beyond Local Learning in Video Autoregression

What it claims

Self Forcing has a "forward–backward gap". A generated chunk shapes later chunks in the forward rollout, but because history is detached, nothing later in the rollout trains how that chunk was produced. Connected Self Forcing (CSF) restores selected future-to-history gradient paths with Shortcut Gradient Replay. It adds no parameters, keeps Self Forcing's detached rollout, and leaves inference unchanged. The authors report gains in imaging quality and background/subject consistency on 60- and 240-second evaluations, with less subject and scene drift. The work comes from XPeng, CUHK and Tsinghua.

How it works

1. Run the standard detached Self Forcing rollout. For each chunk, generate x̂ᵢ from detached history, then write its KV cache Cᵢ by re-encoding x̂ᵢ at the clean context timestep. Save only the states needed to replay each chunk, not the computation graph.
2. Compute the video-level DMD gradient for every chunk, using a real-score model and an online fake-score model.
3. Replay chunks in reverse order (i = N…1). Replay chunk i with a selected historical cache Cⱼ treated as a differentiable leaf, keeping the rest of the history detached. This gives a gradient on Cⱼ straight from chunk i's DMD gradient. It is a "shortcut" because it uses the causal attention read and skips the chunks in between.
4. Backpropagate that gradient through chunk j's KV writer. This updates the writer and yields a gradient on x̂ⱼ, which is then replayed through chunk j's generator.
5. Generation feedback stops at chunk j. KV-writer gradients can flow to earlier writers, but with each writer's latent input detached.

flowchart LR
  A[DMD grad on chunk i] --> B[Replay chunk i, C_j differentiable]
  B --> C[Grad on KV cache C_j]
  C --> D[KV writer of chunk j]
  D --> E[Grad on latent x̂_j]
  E --> F[Replay generator of chunk j]
  F --> G[Stop: no recursion into earlier generation]
  D --> H[Earlier KV writers, latent input detached]

The authors rejected two alternatives:

• Full BPTT is ruled out because of activation memory.
• Chunk-by-chunk recursive relay costs replays that grow with gradient distance, and it makes the signal depend on the Jacobians of the intervening chunks.

CSF does not compute the full cross-chunk gradient. It trades recomputation for memory.

Evidence

• 1D Gaussian toy experiment (deterministic target, so MSE against ground truth is possible; training horizon 20, rollout evaluated to step 124). Self Forcing's validation error plateaus. CSF keeps improving toward Full BPTT and nearly matches Full BPTT's rollout accuracy beyond the training horizon. The authors note the toy is far simpler than video.
• Video: the fetched text claims improvements at 60 s and 240 s but contains no metric values or baseline tables.

Limits

• The fetched text breaks off in the middle of Algorithm 1. There are no quantitative video results, no ablations (for example of the λ_cross weight), and no stated training memory or time cost.
• The base model, code and weights are not given in the fetched text; it only points to a project page.
• CSF recovers only selected gradient paths, not the true full-rollout gradient.
• Concurrent work targets the same gap differently, and a comparison with it is not shown here:
• Self Gradient Forcing trains KV writing only, with context kept stop-gradient.
• Vidu S2's Self-Replay Forcing replays re-noised trajectories.