arxiv.orgpaperOn the Diffusibility of High-Dimensional Latents
This paper studies Representation Autoencoders (RAEs), which let diffusion models generate images in the feature space of pretrained visual encoders. It shows that finetuning encoders for better reconstruction reduces the effective dimensionality of their representations, which makes standard velocity-prediction flow matching inefficient because the model must fit noise directions orthogonal to the low-dimensional signal manifold; switching to x0-prediction (clean data parameterization) instead consistently improves text-to-image generation across multiple strong-reconstruction encoders.
TL;DR Shows that x0-prediction beats velocity prediction for diffusion models trained on high-dimensional latents from finetuned reconstruction-optimized encoders.
Technologies named here
- diffusion models
- flow matching
Techniques named here
- Representation Autoencoders
- Velocity prediction
- x0-prediction
Related in research