Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

What it claims

MF-1 generates both text and images by Flow Matching over frozen encoder embeddings, with no discrete text tokens and no image tokenizer. The 1.6B model scores 0.821 on GenEval and 83.44 on DPG-Bench, which averages to the 82.8 in the summary, and 75.3 averaged across VQAv2, MMBench and POPE after 150B pretraining tokens. The second result is about pretraining value. At a matched total training-token budget for the 1.6B architecture, starting from mixed pretraining instead of a randomly initialized flow backbone raises SeedBench from 31.6 to 62.4 and MMBench from 36.0 to 67.2.

How it works

• Representations are borrowed and frozen. Images go through a frozen SigLIP 2 encoder. Text is split into contiguous blocks of B = 8 tokens, and each block is encoded independently by a frozen T5-small encoder. Decoding uses a pretrained image decoder (Tong et al., 2026) and a separately trained text decoder, and both stay frozen during flow pretraining. Each modality is normalized, kept at its own dimensionality, and then projected into a shared hidden space.
• Hyperchunks. One image forms one visual chunk and keeps its spatial grid. Each 8-token text block forms one text chunk. MRoPE encodes both the chunk order and positions within each chunk.
• Backbone. Each layer applies joint self-attention under a chunk-causal mask, then a modality-specific FFN. Within a chunk, all positions are modeled jointly. Across chunks, the factorization is autoregressive.
• Objective. The path is linear, z_t = t·x + (1−t)·ε. The network predicts the clean endpoint x̂, which is converted to a velocity (x̂ − z_t)/(1 − t) and regressed against x − ε. Each target chunk samples its own timestep.
• Parallel training. The model sees clean and noisy copies of every chunk. A noisy target can attend to earlier clean chunks and to itself, but not to its own clean copy or to later chunks. This gives many chunk losses per forward pass. Packed sequences use sequence IDs in the mask so samples stay isolated.
• One task mixture. Pretraining mixes text-only data, image-only data (unconditional), and paired data in both orders (text→image and image→text). Finetuning for VQA or T2I changes only the chunk ordering and keeps the same loss.

flowchart LR
  P[Clean prefix chunks] --> KV[KV cache]
  N[Gaussian noise] --> F[Flow backbone]
  KV --> F
  F -->|integrate t=0→1, CFG| C{Target modality}
  C -->|text| TD[T5-space → text decoder]
  C -->|image| VD[SigLIP-space → image decoder]
  TD --> A[Append clean chunk]
  VD --> A
  A --> KV

At inference, generation proceeds one chunk at a time. Because finished chunks never depend on future ones, their keys and values are cached, and only the current target chunk is recomputed at each sampling step. Classifier-free guidance (x̂_∅ + γ(x̂_c − x̂_∅)) is applied to both text-to-image and image-conditioned text generation, and the authors report that it helps both.

Evidence

• Benchmark scores are for the 1.6B model at 150B pretraining tokens. The authors describe them as "competitive" with unified models trained on substantially more data, but the fetched text does not give those baselines' numbers.
• Across the 0.6B, 1.2B and 1.6B scales, longer pretraining and larger size "generally" improve language modeling, captioning and T2I.
• Under matched data, optimization and parameter budgets, MF-1 beats fully discrete and hybrid baselines. The comparison tables are not in the fetched text.
• Ablations reportedly favor semantic visual representations (SigLIP-type) and modality-specific FFNs over shared ones. No numbers are given.

How to use it

Code and model are at github.com/hustvl/Multimodal-Flow. The fetched page gives no install commands, checkpoint names or licence. To run the model you need the SigLIP 2 encoder, the T5-small encoder, the external image decoder, and the authors' own trained text decoder.

Limits

• The fetched text ends partway through Section 2.4. The result tables, the details of the baseline comparisons, the appendix (timestep sampling, decoder training) and any limitations section were not available.
• Text quality depends on round-tripping 8-token blocks through T5-small embeddings and a custom decoder. Image fidelity is limited by the SigLIP 2 latent space and the external decoder.
• The authors scope their claims to "the evaluated language–image tasks." The largest model tested is 1.6B, trained on 150B tokens.
• Check the repository for the licence before adopting the model.