NeoMME: an efficient Multimodal-native and Multilingual Encoder

What it claims

H Company's NeoMME is a single bidirectional Transformer, at 260M or 800M parameters, trained from scratch on text and raw image patches with no pretrained vision tower or decoder. Its retrieval fine-tune, NeoMME-Retriever, is the practical payoff:

• Small-model quality: the 260M model scores 0.523 nDCG@10 on ViDoRe v3. That is the best result among evaluated models strictly under 800M parameters, and within 0.002 of ColQwen2.5, which has about 14× more parameters.
• Larger model: the 800M model reaches 0.556, 0.009 behind Vultron Retriever Flash (0.8B).
• Encoding speed: the 260M model encodes about 51 pages/s versus ColModernVBERT's 26 at 2048×2048 on one L40S.

How it works

• Inputs: text uses factorized token embeddings and a 131k-vocabulary BPE tokenizer trained from scratch on multilingual text, code, math and image transcripts. Images are cut into 32×32 patches at their native aspect ratio and size, then passed through a small MLP. Both go into the same encoder.
• Context and attention: the context length is 16,384 tokens, about two 4K UHD images. Most layers use symmetric sliding-window attention. Every sixth layer and the final layer use global attention.
• Encoder stack: grouped-query attention, QK-norm, gated attention, 2D RoPE and squared-ReLU MLPs.
• Pretraining objective: masked discrete diffusion on text only. Text-only examples sample a corruption rate uniformly in [0,1]. Multimodal examples use [0.3,1] with image patches left visible, so heavy masking forces the model to describe the image.
• Pretraining budget: about 524B packed tokens, of which 290B come from text-only examples. The authors note this is small next to ModernBERT's 2T, which is why they chose the NorMuon optimizer for data efficiency.
• Retriever heads: a dense head (mean-pooled, normalized) and a late-interaction head (one 128-d normalized vector per token or patch). The two heads are trained jointly and both are returned from one forward pass.

Evidence

• ViDoRe v1/v2 (nDCG@5): the 260M model beats ColModernVBERT and ColSmol-500M. The 800M model beats ColPali v1.3, which has 3.6× more parameters. Some baseline scores come from MTEB and some from the authors' own runs.
• Storage: a 2048×2048 page produces about 4,200 vectors, roughly 2.1 MB in float32. The ViDoRe v3 average is about 1.5 MB per page. Two measured compression settings on ViDoRe v3:

• Throughput: measured on preprocessed image tensors, with batch size calibrated separately for each model and image size. Both NeoMME sizes are also faster than the compared models at smaller input sizes.

How to use it

The checkpoints are Hcompany/NeoMME-260M-Retriever and an 800M variant, all released under Apache 2.0. The page's install command needs Transformers from main:

pip install -U accelerate "transformers @ git+https://github.com/huggingface/transformers.git@main" "sentence-transformers>=6.0.0"

The API uses NeoMMEProcessor.apply_chat_template(messages, task="query"|"document", ...) to prepare inputs and NeoMMEForRetrieval, which returns .embeddings (late interaction) and .dense_embeddings. Scores are computed with sentence_transformers.util.mean_maxsim and cos_sim.

The authors recommend late interaction by default, for example with NextPlaid. For very large corpora they suggest retrieving with dense ANN first and reranking with late interaction:

flowchart LR
  P[Page images] --> F[One forward pass]
  F --> D[Dense vector]
  F --> L[Late-interaction vectors]
  L --> C[Pool + quantize]
  D --> Q{Very large corpus?}
  Q -- yes --> A[ANN top-k] --> R[Late-interaction rerank]
  Q -- no --> M[Late-interaction search directly]
  C --> R
  C --> M

A demo Space is available at tonywu71/neomme-retriever-demo.

Limits

• Fine-tuning heads separately: Sentence Transformers v6 supports only one retrieval head per model. The page ships separate dense and late-interaction checkpoints for it. Training both heads together requires NeoMMEForRetrieval with a custom Trainer.
• Unreleased Transformers support: the integration currently needs Transformers installed from git main.
• Self-run baselines: some baseline numbers come from the authors' own evaluations rather than MTEB.
• Narrow throughput setup: speed was measured on preprocessed tensors, one GPU type (L40S) and per-model tuned batch sizes. End-to-end indexing speed may differ.
• Single downstream task: visual document retrieval is the only fine-tune reported. The backbone has not been evaluated here on classification or token labeling.
• Limited compute: the authors describe the project as a side project done with limited time and compute.