Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

What it claims

Nemotron-pretraining-Indic-v1 is a synthetic corpus of about 2.8T tokens (2,601.0B native-script, 224.6B romanized; Gemma-4 tokenizer) translated from English Nemotron-CC-v2 sources into the 22 constitutionally listed Indian languages. NVIDIA, IIT Madras/AI4Bharat and Bodhan AI built it together. The post states that the data is "releasing soon." It gives no repository ID and no licence. The generation recipe is available now as the Nemotron Bharat translation plugin for NeMo Data Designer.

How it works

There are two workflows, split by language:

• Twelve languages, native and romanized: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, Urdu.
• Ten languages, native script only: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, Sindhi.

flowchart TD
  A[English doc, Nemotron-CC-v2] --> B{Language group}
  B -->|12 langs| C[Gemma-4-31B-it translate]
  C --> D{MQM-style judge ≥75?}
  D -->|no| E[Feedback repair] --> D
  D -->|yes| F[Native-script output]
  F --> G[Romanize subset, check vs native]
  B -->|10 langs| H[Indic Translate]
  H --> I{Gemma judge ≥75?}
  I -->|no| X[Dropped, no repair]
  I -->|yes| J[Native-script output]

• Translation: whole documents are translated in one pass. Prompts mark code, equations, URLs and formatting as content to preserve rather than translate.
• Repair yield: the repair loop recovered 438.4B tokens, which are included in the 2.8T total.
• Romanization: romanized text is converted from accepted native-script output, not translated directly from English. In an initial human comparison, this two-step route produced fewer hallucinated additions and less residual English.
• Ten-language adaptations: these use Bodhan AI's Indic Translate model, chosen because of long-document and script-stability problems. Long Santali documents are translated in overlapping windows and then reassembled. Manipuri is translated into Bengali script and then converted to Meitei Mayek.

Evidence

The model was Nemotron 3.5 Super VL, with continued pretraining (CPT) on a 100B-token budget:

• Mixture: 10B Indic tokens, taken from the twelve-language subset plus natural Indic text. The other 90% kept the earlier training mixture's proportions.
• Results: compared with the checkpoint just before CPT, scores rose 4.98–9.82 points on MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy and Indic GSM8K. GSM8K excludes Odia because evaluation exceeded the serving context limit.
• What this does not show: the authors say the comparison does not isolate the synthetic corpus's contribution. The companion report's controlled ablations used only first-pass accepted twelve-language data, so they exclude the repaired data and the ten-language data.

The released model was also compared with Nemotron 3 Super, which includes the effect of post-training:

• It is higher on five of six task benchmarks.
• IFEval 1.5 instruction following dropped 2.04 points.
• All four IndiVibe dimensions improved, but these are preliminary conditional preference scores, not head-to-head win rates.

How to use it

Configurations are named by language, script and region, for example hi-Deva-IN (Hindi in Devanagari) and hi-Latn-IN (romanized Hindi). The page's loader leaves the repository ID as a prompt because the ID is not yet published:

from datasets import get_dataset_split_names, load_dataset
dataset_id = input("Repository ID from the dataset card: ").strip()
config = "hi-Deva-IN"
splits = get_dataset_split_names(dataset_id, config_name=config)
print("Available splits:", splits)
data = load_dataset(
    dataset_id, name=config, split=splits[0], streaming=True
)
print(next(iter(data)))

Practical guidance from the post:

• Separate corpus size from training budget. Token counts include translations and script variants of the same source documents. They are not unique-document counts or a training budget.
• Split held-out data by source. Records derived from one source exist across languages and scripts, so held-out splits must account for that to avoid leakage. Source IDs are retained for this.
• Track mixture shares and retention. Record the natural/synthetic and native/romanized shares at a fixed token budget, and track English capability retention.
• Pin a revision once the dataset is released.
• Filter separately from generation. The plugin returns candidates and scores. Which records enter the corpus is decided by your own filtering policy.

Limits

• The data is not released yet, and no licence is stated on the page.
• Quality judging is mostly automated. In the twelve-language workflow, the translator also acts as the judge.
• Human review is sampled, and its coverage varies by language. The ten-language data needs broader native-speaker validation.
• The ten-language workflow has no repair step.
• Deduplication is by source ID only, with no semantic deduplication.
• Translated text does not supply local or cultural knowledge. The authors recommend mixing it with natural Indic text.