Skip to content

Back to the catalog

Self-correction Optimization for Interleaved Multimodal Generation
arxiv.orgpaper

Self-correction Optimization for Interleaved Multimodal Generation

This paper introduces Self-Correction Optimization (SCO), a training-free method that improves consistency in interleaved image-text generation from multimodal large language models. SCO treats the classifier-free guidance update as a reference signal and applies minimal corrections under two constraints—a new-event constraint for temporal consistency and a state-preserving constraint for visual-subject coherence—avoiding the computational cost of additional training with augmented data. The authors show improvements on interleaved multimodal generation benchmarks and extend the method to video generation tasks like robot manipulation and long-horizon handcrafting.

TL;DR A training-free self-correction technique for MLLMs that improves temporal and subject consistency in interleaved image-text and video generation.
Open original ↗
Document

What it claims

SCO is a sampling-time change to classifier-free guidance (CFG) for interleaved image–text generators. It needs no extra parameters or training. Standard CFG feeds the full condition (history plus current instruction) into one guidance direction. That direction cannot separate what should change from what should stay. The paper names two failure modes that compound over rounds:

  • under-realization: the model stops at an intermediate state, over-relying on the last image;
  • subject drift: identity, layout or style changes across image slots.

SCO keeps the CFG update as a reference and nudges it, as little as possible, toward two separately estimated directions. It is evaluated on OpenING (5,400 human-annotated instances, 23 meta-topics, 56 tasks) and ISG-Bench (1,150 samples, 8 scenarios, 21 subtasks), and is extended to video for robot manipulation and long-horizon handcrafting.

How it works

At each selected sampling step, the frozen generator is evaluated four times at the same latent state. Standard CFG uses two of these.

flowchart LR
  U[Unconditional v_u] --> REF[CFG reference δ_ref]
  F[Full: history + event] --> REF
  F --> NEW[a_new = v_f − v_counter]
  C[Counterfactual: history + pre-event state e∅] --> NEW
  F --> ST[a_state = v_f − v_event]
  E[Event only: no history] --> ST
  REF --> P[Bounded 2-D projection]
  NEW --> P
  ST --> P
  P --> OUT[δ* replaces CFG update]
  1. New-event direction. This is the full-condition output minus a counterfactual output. The counterfactual keeps the same history but swaps the instruction for e∅, a description of the state just before the event. The constraint requires the update's projection onto this direction to be at least λ_new(τ)·‖a_new‖.
  2. State-preserving direction. This is the full-condition output minus an event-only output, where the history is dropped. The constraint has the same form, with threshold λ_state(τ)·‖a_state‖. It acts in update space, so no pixel correspondence is needed.
  3. Joint projection. SCO finds the update closest to δ_ref that satisfies both half-space constraints. - The two directions are not orthogonal. The cosine between them signals whether they cooperate or compete. - Applying the two corrections one after the other can undo the first, so they are solved jointly. - Each correction coefficient is capped: 0 ≤ µ_new ≤ M_new and 0 ≤ µ_state ≤ M_state. This is equivalent to a soft-constraint penalty. Any residual violation shows that a budget ran out or the constraints conflicted. - With only two coefficients, the active sets are enumerated directly, so no generic QP solver is needed.
  4. Schedule. λ_new = α0 for 0 ≤ τ ≤ τ_end, and zero after that. λ_state = s_state·λ_new. The correction therefore applies only early in sampling.
  5. Video. Each temporal position is treated as an image slot. A GPT-5 planning stage splits the prompt into ordered events and assigns frames to them. SCO is then applied to each frame-indexed component, including when frames are generated jointly. Details are in Appendix A.1, which is not in the fetched text.

Evidence

The fetched text stops at the dataset description. No quantitative results are present. There are no metric tables, baseline numbers or ablations of the two constraints or the joint-versus-sequential projection. Figure 2 shows qualitative before/after pairs for three tasks:

  • embodied next-frame forecasting;
  • 5-step novel view synthesis (rotating a llama's head);
  • 5-style art transfer.

Limits

  • No numbers to assess. The claimed "significant improvements" cannot be checked from this text. The base generator used as "Baseline" is not named in the fetched portion.
  • Inference cost. Per corrected step, SCO needs four model evaluations instead of CFG's two. The step only runs up to τ_end.
  • Extra prompt per round. Each round needs e∅, a pre-event state description. The fetched text does not say how it is produced for benchmark items.
  • Unreported hyperparameters. Values for α0, s_state, τ_end, M_new and M_state are not given here.
  • Video depends on GPT-5. The video extension relies on an external GPT-5 planner.
  • No code or licence. No code release or licence is mentioned.

Technologies named here

Techniques named here

  • Self-correction optimization
  • New-event constraint
  • State-preserving constraint

Related in research