Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback

What it claims

In MMDiTs (SD3/3.5, FLUX, HunyuanVideo), text tokens attend to image tokens. This image-to-text (I2T) path carries the current image's content, including its mistakes, back into the text stream that later blocks use for conditioning.

The causal evidence is a transplant test on 400 success/failure pairs:

• Swapping a successful run's I2T messages into a failed run rescues 67.5% of failures.
• The reverse swap breaks 60.5% of successes.
• Norm-matched random messages flip only 1.3%.

Suppressing I2T feedback, in either the guidance form (SFG) or the in-place form (iSFG), improves compositional alignment. Without CFG, on FLUX, GenEval rises from 0.3604 to 0.6853.

How it works

Joint attention splits each block's output into a within-modality term plus a cross-modal message: O_t = A_t→t V_t + B_i→t (the I2T message) and O_i = A_i→i V_i + B_t→i (the T2I message). All edits scale these messages after softmax and value aggregation, before output projection. Attention logits and joint normalization are untouched, and weights stay frozen.

SFG adds an auxiliary forward pass with edit strength u < 0:

• SFG-X: the I2T message is multiplied by (1−u) > 1, strengthening feedback.
• SFG-S: the T2I message is multiplied by (1−|u|), weakening conditioning. The page requires |u| ≤ 1, since larger values flip the sign.

The sampler then guides away from this auxiliary prediction. Unlike PAG and SEG, the negative branch strengthens a native pathway instead of degrading the model.

flowchart LR
  C[clean conditional pass] --> V[combined velocity]
  P["aux pass: I2T up, T2I down<br/>(first 20% of steps only)"] -->|"ω · (clean − aux)"| V
  U[unconditional pass] -->|"(s−1) · CFG term"| V

The combined update is v̂ = v_clean + ω(v_clean − v_pert) + (s−1)(v_clean − v_uncond). At s = 1 the unconditional pass is skipped.

iSFG applies the edits directly inside the ordinary conditional pass. The I2T message is multiplied by (1+γ) with γ < 0, and the T2I message by (1+γ_S) with γ_S > 0. Only the conditional branch is edited, so there are no extra model evaluations.

Cost. The auxiliary branch runs only during the first 20% of K steps. That gives about 1.2K evaluations without CFG and 2.2K with CFG. The window choice comes from timing data: transplants in steps 1–8 of 40 give 43% rescue, about two-thirds of the full-schedule 67.5%, and late transplants rarely change the outcome.

Evidence

Table 1 compares methods without CFG (cfg = 1) at 1024²:

• FID (SD3M, 5k COCO images): SFG reaches 24.21, against 41.76 for the base model, 26.00 for S2-Guidance and 32.49 for CFG at s=7.5.
• Ablation: almost all of the gain comes from the I2T component. SFG-X alone beats every baseline, while SFG-S alone trails PAG and SEG on GenEval.
• Relation to CFG: SFG corrections stay nearly orthogonal to CFG throughout sampling.
• Protocol: GenEval uses 553 prompts × 6 seeds (3,318 images). HPSv2.1 and ImageReward use 3,200 images (800 prompts × 4 seeds, four styles). Methods share prompts and seeds within each configuration.

How to use it

Code: https://github.com/yyc1215lll-creator/SFG.

Implementation means hooking each joint-attention block to scale B_i→t and B_t→i between value aggregation and output projection. If you cannot afford any extra evaluations, iSFG is the zero-NFE option.

Limits

• Truncated text. The fetched page ends partway through the results. The CFG-combined tables, the TACA and Prompt Reinjection comparisons, the HPSv2.1/ImageReward tables and the video VBench numbers are not in it.
• No hyperparameters. Values for u, ω, γ and γ_S are deferred to the appendices and not shown.
• Narrow analysis set. The mechanism analysis covers only SD3.5M and FLUX, and FID is reported only for SD3M.
• Some disruption. Suppression can still disrupt initially successful outputs, though less often than it rescues failures.
• No licence stated for the code.