Adversarial Training for Pixel Diffusion

What it claims

Adding a hinge GAN loss on the predicted clean image to an already-converged pixel text-to-image model, while keeping its diffusion or flow-matching loss, improves every reported metric on two backbones. It needs no distillation, no reward model and no change to step count or sampler. On DeCo:

Real COCO images have α ≈ 2.19. The same procedure applied to two latent models, PixArt-α and SANA, gives no comparable joint improvement.

How it works

• Generator loss. The original objective plus L_G = w_gan · E[−D(h(ŷ0))].
• Discriminator loss. The discriminator minimises the hinge loss ½E[relu(1−D(h(y0)))] + ½E[relu(1+D(h(ŷ0)))].
• Recovering ŷ0.
• For velocity models (DeCo, SANA), ŷ0 = y_t − σ_t·v_θ.
• PixelGen predicts x̂0 directly.
• PixArt-α inverts its ε-prediction.
• The map h. For the pixel models h is the identity. For the decoded-RGB latent variants, h is the frozen VAE decoder.
• Timestep gate. The GAN term is applied only where the signal fraction α_t = 1−σ_t ≥ τ. The exact τ values are in an appendix that is not on this page.
• Discriminators. Frozen DINOv2, DINOv3 or SigLIP backbones with text-conditioned projection heads.
• Data. Fine-tuning runs from public checkpoints on BLIP3o-60k, the dataset DeCo and PixelGen were originally trained on.
• Controls. Each GAN run is paired with a no-GAN SFT control that has the same starting checkpoint, data and step count.

flowchart LR
  A[noisy input y_t] --> B[pretrained denoiser]
  B --> C[predicted ŷ0]
  C --> L1[original diffusion/FM loss]
  C --> G{α_t ≥ τ?}
  G -- no, high noise --> L1
  G -- yes --> H[h: identity for pixel / VAE decoder for latent]
  H --> D[frozen-backbone discriminator]
  D --> L2[hinge GAN loss]
  L1 --> T[total generator loss]
  L2 --> T

Evidence

All figures are on COCO-30k, except DPG Score, which is on DPG-Bench. Each comparison is against the matched no-GAN SFT control.

• PixelGen.
• FID 33.94 → 33.20
• recall 0.319 → 0.403
• DPG 78.4 → 80.8
• CMMD 0.762 → 0.725
• MANIQA 0.645 → 0.723
• DeCo, other metrics.
• pFID 27.91 → 24.38
• CMMD 0.836 → 0.736
• TOPIQ 0.711 → 0.768
• MANIQA 0.636 → 0.712
• Spectral power, averaged over 30,000 images per model.
• DeCo: mid band 4.9% → 7.5%, high band 2.1% → 3.9%.
• PixelGen: mid band 6.6% → 8.9%, high band 3.9% → 6.4%.
• On DeCo, absolute high-frequency power rises by +0.34 dex.
• SANA and PixArt barely change. PixArt's high band goes from 2.1% to 1.9%.
• Not just sharpening. An unsharp mask tuned to match the GAN's high-frequency spectrum reaches FID 32.3 at best, against 28.59 for the GAN.
• Not memorisation. Mean DINOv2 nearest-neighbour similarity to the ~60k training images is unchanged at 0.586. The maximum falls from 0.943 to 0.929.
• Latent probe. Perturbing inputs to the frozen PixArt VAE decoder gives a 3.5–11× weaker high-frequency response in the decoded image.
• Perceptual-loss alternative. An LPIPS+DINO perceptual objective, as in PixelGen, also adds high frequency. However, it causes a desaturated, low-contrast shift and worsens FID and prompt alignment.
• DPG trajectories. Starting from the official pre-SFT checkpoints (DPG 81.4 for DeCo, 79.4 for PixelGen), the GAN curves rise above the perceptual-loss curves over roughly 30k optimizer steps.

Limits

• Latent models. The negative result covers only the tested PixArt-α and SANA configurations. The authors frame decoder access as "an important mechanism", not a proof that the method cannot work in latent space.
• Scope. Only two pixel backbones were evaluated, both fine-tuned on a ~60k-image dataset.
• Trade-offs. Discriminator choice, GAN weight and gate threshold shift the metric trade-offs. Those ablations are not in the fetched text.
• Missing material. The page excerpt is truncated partway through the perceptual-loss comparison. It mentions no code, weights, licence or training compute.