Labadain-MedQA: A Tetun Medical QA Dataset for Fine-Tuning and Evaluating Large Language Models

What it claims

Labadain-MedQA is a Tetun dataset of 39,077 medical QA pairs: 33,135 from MedAlpaca's Medical Flashcards and 5,942 from WikiDoc Patient Information. It ships with 325 Tetun-translated USMLE multiple-choice questions for evaluation. Both parts were machine-translated with Gemini 2.5 Flash. On the USMLE set, Gemma 4 12B fine-tuned with QLoRA improved mean accuracy over its base model by 10.79% (relative) with 4-bit NF4 inference. After merging the LoRA adapters into the BF16 base model, the gain was 10.42%. The datasets, fine-tuned models, human reference translations and annotation data are released at https://github.com/labadain/labadain-medqa.

How it works

Translation uses two calls to Gemini 2.5 Flash. The first call translates each QA pair, guided by ten English-to-Tetun medical translation examples. The second call revises orthography, using Portuguese–Tetun transformation rules and a curated list of common Tetun errors with their corrections. The authors traced outputs in LangSmith, inspected them by hand (the first 10, then up to 50) for terminology, coherence and Tetun usage, and refined both prompts in response.

flowchart LR
  A[English QA pairs] --> B[Call 1: translate, 10 examples]
  B --> C[Call 2: orthographic revision]
  C --> D[Manual inspection]
  D -->|issues found| E[Refine prompts]
  E --> B
  D -->|accepted| F[Tetun dataset]

Fine-tuning used QLoRA on six models: Gemma 4 E2B, E4B and 12B, plus LLaMA 3.1 8B, LLaMA 3.2 1B and 3B. The USMLE questions were never used for training.

Evidence

The authors checked translation quality on 50 QA pairs (100 documents), 25 pairs from each collection. A professional native Tetun translator produced reference translations. Two native Tetun-speaking healthcare professionals then rated the human and LLM translations on medical accuracy and fluency: 4.43/5 for human versus 4.02/5 for LLM (Wilcoxon signed-rank, p < 0.001). Automatic scores against the references were computed with BLEU-4, chrF++ and ROUGE-L.

On length, mean question length is 13.74 words in English and 13.33 in Tetun. Mean answer length is 56.44 words in English and 53.31 in Tetun, and Tetun runs shorter mainly on long answers.

How to use it

Each MedAlpaca-derived record has four fields: input_en, input_tdt, output_en and output_tdt. English is kept alongside Tetun, so the data also works as a parallel corpus. Each USMLE item has answer fields A to I, with at most eight filled; most questions have five options and unused fields are null. The USMLE split has 94 Step 1, 109 Step 2 and 122 Step 3 questions.

Limits

• Dropped data: 820 Flashcards pairs were dropped for missing values or parsing errors.
• Translated evaluation set: the USMLE set was translated with the same Gemini pipeline and was not human-translated. Benchmark scores therefore inherit any translation errors.
• Small human check: only 100 documents received human review.
• Truncated page: the fetched text ends before the automatic metric scores, the absolute accuracies and the results for the other five models. The gains above are relative only.
• No licence: the fetched text states no licence for the data or the models.