Know, then Draw: A New Learning Paradigm for Visual Generation

What it claims

Know, then Draw (Yucheng Mao and colleagues at UC San Diego and Adobe Research, arXiv, October 2026) splits text-to-image training into two stages. A text-conditioned autoregressive (AR) model first learns to predict discrete semantic image tokens. A pixel-diffusion renderer is then post-trained on top of the frozen AR backbone. The 3B model reaches 0.86 on GenEval and 87.78 on DPG-Bench with about 6,600 H100 GPU-hours. The authors estimate this is about 1/48 of Z-Image-Base's training compute, at comparable DPG-Bench performance.

How it works

• AR pretraining. A frozen visual tokenizer maps each image to discrete tokens, and a frozen text encoder supplies conditioning. The AR model trains on next-token cross-entropy over a random permutation of spatial positions, following RandAR, with position-instruction tokens.
• Diffusion post-training. The AR backbone is frozen. New trainable modules reuse its frozen attention Q/K/V/O projections. These modules are an input adapter for noisy patches, timestep embeddings, a diffusion FFN with AdaLN, and a linear pixel readout. The renderer predicts the clean image, converts it to velocity, and trains on a velocity MSE (pixel-space flow matching, logit-normal timesteps with a resolution-dependent shift).
• Conditioning switch. During post-training, the renderer conditions on tokens from the ground-truth image. At inference, it conditions on tokens the AR model generates from the prompt.

Three tokenizers were compared, all with IBQ codebooks:

• Recon is the Emu3.5 tokenizer, which reconstructs pixels.
• Semantic-Recon quantizes frozen Qwen2.5-VL vision features and trains a decoder with reconstruction loss.
• Semantic-LM trains the codebook only through next-token prediction inside a frozen Qwen2.5-VL. It has no image decoder.

Compute is split between the stages using fitted laws of the form L(N, D) = E + A(N/N0)^−α + B(D/D0)^−β, with C = 6ND as the compute proxy. For post-training, that proxy excludes the frozen AR forward pass.

Evidence

The objective comparison used final generators of about 400M parameters and the same number of training images. The two-stage routes paired a 250M pretrained backbone with 150M rendering parameters.

The AR route scores 40.00 on counting, against 33.75 for pixel diffusion. It reaches 64.8 GenEval after 100k post-training updates, above pixel diffusion trained for 400k iterations (61.2).

The tokenizer comparison shows that reconstruction quality and final quality diverge:

Recon is the only tokenizer whose scores fall after diffusion post-training. At this scale, Semantic-LM lands slightly below Semantic-Recon, so its advantage is dropping the decoder rather than a higher score.

At matched model and data scale, the fitted asymptotic loss E for diffusion post-training is lower than for pixel diffusion trained from scratch. The authors read this as AR pretraining raising attainable quality, not only speeding up convergence. The fetched text does not include the fitted exponents or the chosen compute split r.

Limits

• The fetched text ends partway through Section 3.2. The scaling-law fits, the 3B model's tokenizer choice and data, and the comparisons against other models are missing.
• No weights, code or licence are mentioned in the fetched text.
• The 48× figure compares against an estimate of Z-Image-Base's compute, not a reported number.
• The 6ND proxy for post-training leaves out the frozen AR forward pass, so it understates the true cost of the rendering stage.
• The controlled comparisons ran only at about 400M parameters. The authors qualify the AR-over-LDM result as holding "under the evaluated settings".