A Language-Informed Loss for Visual Generation

What it claims

LILIE is a 32M-parameter invertible image encoder whose feature space is shaped by caption prediction. Its L2 distance serves as a training loss for pixel-space text-to-image models. On GPIC at 256×256, a 1.1B JiT-T2I-g reaches gFD-DINOv2 53.01 with Pixel+LILIE loss, against 66.84 with pixel loss alone. The summary does not say that, in the same table, plain LPIPS added to pixel loss does slightly better than LILIE (48.65 vs 53.01). LILIE's distinct result is as a standalone loss: it trains a working generator (gFD 55.78), while DINOv2, SigLIP 2 and LPIPS used alone collapse or degrade badly.

How it works

Stage 1 trains the encoder. Each image passes through the encoder (improved TARFlow architecture from BiFlow, used for invertibility rather than density modelling). The features are rescaled to a per-image RMS of 1/2 and corrupted with additive Gaussian noise. SmolVLM-256M with rank-64 LoRA then predicts the GPIC caption. Caption cross-entropy is the only loss, and it updates the encoder and LoRA jointly. An EMA of the features' second moment (m2) is tracked. Afterwards the VLM is discarded. The RMS target matches natural-image RMS in GPIC and stops the encoder from inflating features to beat the noise.

Stage 2 trains the generator. The frozen encoder, divided by a fixed 2√m2 so it stays a bijection, is applied to the predicted and target images. The resulting squared error is added to JiT's x-prediction flow-matching loss with weight λ.

flowchart LR
  P[Predicted image] --> PX[Pixel L2]
  T[Target image] --> PX
  P --> E[Frozen LILIE encoder]
  T --> E
  E --> LL[LILIE-space L2]
  PX --> S[w t · Pixel + λ·LILIE]
  LL --> S

The generators condition on a frozen Qwen3-1.7B text encoder, with captions dropped for 10% of samples to support classifier-free guidance.

Evidence

All runs use GPIC Full (100M image-caption pairs), one epoch, batch 256, 390,600 steps. Evaluation is on GPIC Test-50K with 50 Euler steps, and CFG is chosen on Validation-50K.

• JiT-T2I-XL (700M): Pixel 80.24, LILIE 68.88, Pixel+LILIE 65.86.
• JiT-T2I-g (1.1B), standalone losses: DINOv2 3124.01, SigLIP 2 3161.15 (recall 0.000), LPIPS 158.45, LILIE 55.78.
• 1.1B, combined with pixel loss: Pixel+DINOv2 47.88, Pixel+LPIPS 48.65, Pixel+SigLIP 2 52.55, Pixel+LILIE 53.01.
• 1.1B at 781,200 steps: Pixel 49.20, Pixel+DINOv2 38.29, Pixel+LPIPS 39.35, Pixel+LILIE 40.18, Pixel+SigLIP 2 40.27.

The DINOv2 and SigLIP 2 rows are marked non-compliant with the GPIC protocol because those backbones are distilled from larger models. Among compliant combined losses, Pixel+LPIPS beats Pixel+LILIE at both training budgets.

How to use it

Code, encoders and generators are listed at suchiragarwal.com/lilie. No install commands or licence appear in the fetched text. The default encoder, LILIE-S, uses K=4 blocks, 4 layers per block, bottleneck dimension 384 and patch size 16. Training it took about 40 hours on TPU v5e-32. Training JiT-T2I-XL took about 27 hours and JiT-T2I-g about 40 hours, both on TPU v6e-16.

Limits

All results are at 256×256 on a single dataset (GPIC), at one or two epochs. The fetched text is truncated partway through Section 4. The ablations, the value of λ, the noise level σ and any stated limitations section are therefore not visible here. The page gives no licence for the released weights.