Saidzoda-Tajik-E5-Large: Fine-Tuning a 560M Embedder for Tajik Retrieval

What it claims

Saidzoda-Tajik-E5-Large is a full fine-tune of multilingual-e5-large-instruct (560M, XLM-RoBERTa-large) for Tajik passage retrieval. It was trained in about 56 minutes on one H100. On LLM-generated questions over held-out Tajik laws and tax documents, it ties Giga-Embeddings-instruct-3B used as released. When the 3B and 8B models are LoRA-tuned on the same pairs, they pull clearly ahead on the human-written Belebele Tajik set. The authors (Saidzoda AI Lab, Dushanbe) also report that the shared XLM-RoBERTa tokenizer maps 4 of the 12 Tajik-specific letters (ҷ, ӣ, ӯ and the capitals Ҷ, Ӣ, Ӯ) to a single unknown token. This affects multilingual E5, BGE-M3 and other models built on it.

How it works

The authors' own Tajik fine-tune of Qwen3.6-27B writes questions for passages from four sources: 72,920 law chunks, 23,282 tax chunks, Tajik Wikipedia, and web and literature text. A second, independent call to the same model checks each question, and only checked questions are kept. The data totals 235,140 pairs (Table 1). Replay data makes up 36% of it: English Natural Questions (49,987), Russian MIRACL (4,664) and Russian ru-HNP (29,846). A further 15,744 synthetic customer-service pairs are not released.

Hard negatives are mined with BGE-M3. One is drawn at random from ranks 10 to 50, and it must come from a different law, document or article, must not be the positive, and must score below 0.95 times the positive (NV-Retriever margin). 71% of in-domain pairs got a negative. The rest, and all replay pairs, use in-batch negatives only.

Training settings:

• Loss: cached multiple-negatives ranking loss (sentence-transformers), global batch 512 with mini-batches of 64, scale 20.
• Batches: each batch comes from a single domain and never repeats a text.
• Schedule: learning rate 2e-5, linear with 5% warm-up, one epoch (466 steps), max 512 tokens, seed 42.
• Precision: fp32 with TF32.
• Instruction: the original E5 web-search query instruction is kept at train and query time.

Evidence

Results are recall@10 with 95% bootstrap intervals. The test splits hold out whole units, chosen by SHA-1 of the title: 1,978 laws, 22 tax documents and 750 articles. Each test corpus also includes all non-held-out passages as distractors.

Russian and English Belebele stay unchanged (98.6 and 98.9).

Comparisons on law and Belebele Tajik:

• Zero-shot: Qwen3-Embedding-8B scores 76.3 on law and EmbeddingGemma-300M 49.3, even though their tokenizers cover all Tajik letters. Giga-Embeddings-instruct-3B scores 92.2 on law and 97.1 on Belebele Tajik.
• LoRA on 40% of the pairs: Giga-3B reaches 94.6 on law and 97.8 on Belebele. Qwen3-8B reaches 92.8 and 98.2. Their Belebele intervals ([96.8, 98.7] and [97.3, 99.1]) do not overlap the 560M model's [93.9, 96.6].
• BGE-M3 with the same recipe: law gains 14.0 points, tax 13.8 and Belebele Tajik 8.4 (non-overlapping intervals). Russian and English drop by under one point, so the recipe is not tied to E5.

Latency is about 12 ms per query on an H100, against 35 to 38 ms for the 3B and 8B models. These timings come from a shared machine.

Limits

• Generator bias. Law, tax and Wikipedia test questions come from the same generator as the training questions. The authors treat those scores as upper bounds and note that this favours their model over the zero-shot baselines.
• Small or unconfirmed gains. Belebele Tajik is the only human-written test. Its gain has touching intervals, and no paired test was possible because per-question ranks were not stored. The tax set has only 109 questions.
• bfloat16 collapse. In bfloat16, the logged loss hit exactly zero at step 8 and stayed there, at both scale 33.3 and scale 20. A non-finite gradient norm appeared one step earlier. fp32 trained normally.
• Failed ablations. Folding Tajik letters into the Russian ч, и and у lowered law recall to 78.0. WiSE-FT interpolation (0.7 tuned + 0.3 base) scored 93.2 on law and 92.7 on tax, slightly below the final model on both.
• Missing checks. There was no run without replay data. Slight recall@1 gains on Kazakh, Uzbek and Kyrgyz Belebele were not tested against noise.
• Not in the page text. The fetched text gives no weights location, licence or release status for the model or the benchmark.