Large Flow Language Models

What it claims

Lafa is a continuous flow language model from UC Berkeley, Amazon, KAIST and Carnegie Mellon (arXiv, 20 October 2026). It is built by adapting a pretrained autoregressive (AR) model, not by training from scratch. The 7B model is compared against an AR SFT baseline of Qwen2.5 7B. Lafa 7B reaches 2.02 to 2.54 times that baseline's throughput at matched or better accuracy. The authors also place it ahead of MARS, DeepSeek-MTP, Efficient-DLM, Dream and Fast-dLLM v2 on the same quality and speed plot. Adaptation uses under 30B tokens. The authors contrast this with comparable discrete diffusion adaptations that use up to 500B tokens. Joint flow training also raises the model's AR accuracy above matched AR SFT, which the authors attribute to a multi-token prediction effect.

How it works

Text is split into blocks of size B. Each block is embedded with the model's own vocabulary embeddings (dimension d, not one-hot over |V|) and generated by flow matching from Gaussian noise.

• Tokenwise time. Each block position has its own schedule τᵢ(t). This schedule is learned online as the inverse CDF of that position's normalized denoising loss across noise levels. The learned schedules denoise earlier positions first, although no left-to-right bias is imposed by hand.
• Denoiser, not velocity. The model predicts clean tokens by classification. The velocity is recovered from the predicted mean embedding by a closed-form identity, which is proved in Appendix A.1.
• Adaptation changes. adaLN time conditioning is initialized to identity, and prefix positions always receive t = 1. Attention is bidirectional inside the noisy block and causal on the prefix, so prefix KV caches never depend on the block state. Embeddings are shared and trained, but each keeps its pretrained L2 norm and learns only its direction. Ablations (Table 3) favor this over frozen or unconstrained embeddings, and favor bidirectional over causal block attention.
• Loss. The loss is the sum of AR cross entropy on the prefix, denoising cross entropy, and embedding squared error. All blocks of a sequence train in one forward pass by pairing the clean sequence with a noisy replica.

flowchart LR
  A[Clean sequence] --> F[Shared transformer]
  B[Noisy block replica] --> F
  F --> C[AR cross entropy]
  F --> D[Denoising CE + embedding MSE]

• Self-conditioning reuses the previous Euler step's prediction, so it costs no extra forward pass at inference.
• Prefix guidance, which is analogous to classifier-free guidance, adds one prefix-free pass per step. That pass covers only the B block tokens.
• Flow map distillation extends simplex-valued two-time denoisers to asynchronous flows. The student starts from the frozen teacher and is trained with semigroup consistency and KL losses. It also distills the teacher's prefix AR predictions, keeps prefix caching, and allows one-step block generation.
• Self-speculative decoding drafts a block with a single denoiser evaluation from noise, without guidance. The same model's AR head then verifies the draft left to right.

How to use it

The page links code at github.com/lafa-gh/large-flow-lms and weights at huggingface.co/lafa-hf/models. Models were trained at 0.5B, 7B and 8B. The 7B model is the one compared against Qwen2.5 7B. The page does not name the 0.5B and 8B base models, and it gives no install commands or licence.

Limits

The fetched text stops partway through Section 2.3. Per-benchmark tables, block size, step counts, verification acceptance rates, hardware and the conditions behind the 2.02 to 2.54x range are not shown. Evaluation is described only as covering instruction following, math and coding. The method assumes a pretrained AR model to start from. The authors note that earlier flow language models were shown only at small scale, and that a prior scaling attempt hit training inefficiency.