An Empirical Study of Recent Large Language Models for Knowledge Graph to Text Generation on Standard Benchmarks

What it claims

Recent prompted LLMs have cut the gap with a fully fine-tuned T5-large on KG-to-text from tens of BLEU points (in earlier published zero-shot results) to a few. The benchmarks are WebNLG17, WebNLG20 (English) and DART. The prompted models reach parity on neural metrics. Fine-tuning still leads on BLEU, BARTScore and PARENT.

The second finding concerns faithfulness. By human annotation, fine-tuned and prompted systems have a faithfulness problem in a similar share of outputs, but the error types differ. Fine-tuned models corrupt entities and omit triples. Prompted models add unsupported elaboration. The LLM judges used here detect omissions and contradictions but do not detect elaboration, so judge scores understate the prompted models' main failure.

How it works

The models range from 270M parameters to a 550B sparse model:

• Fine-tuned baselines: T5 Small, Base and Large, fully fine-tuned with the Shi et al. (2023) pipeline.
• LoRA-adapted encoder-decoder: T5-Gemma2 at 4B-4B. Checkpoints at 270M-270M and 1B-1B are also listed.
• Prompted models: Gemma-4 (E4B-it, 26B-A4B MoE, 31B-it), Granite-4.2 (3B, 8B, 30B), Nemotron-3 (Nano-30B-A3B, Super-120B-A12B, Ultra-550B-A55B, a hybrid of Mamba-2 and MoE layers), and GPT-5 nano, mini and full.

All systems get the same graph linearization, with triples separated by <H>, <R> and <T> tokens.

T5 training used learning rate 3e-5, batch size 8 and up to 30 epochs with early stopping (patience 3). Sequences were capped at 384 tokens and decoding used beam size 3, on a single A40 or A6000 GPU per model. T5-Gemma2 LoRA used r=32, α=64, dropout 0.05 and learning rate 5e-4 on q_proj and v_proj. That setting was chosen by a sweep over r in {8, 16, 32} and four learning rates, with 3 seeds each.

Prompted models used one elaborate prompt (the paper's Table 10), shared across all models, with temperature 0.0 and max 384 tokens. Thinking was set to off, and GPT-5 reasoning to "minimal". Few-shot examples were retrieved with E5-base-V2 embeddings of the linearized graphs, after removing training items identical to test items. Three retrieval strategies were compared:

• cosine similarity;
• Maximal Marginal Relevance;
• triple-count bucketing: the n-triple subset for 1-shot, n and n+1 for 2-shot, and n−1, n and n+1 for 3-shot.

Evidence

All metrics use a multi-reference protocol. BLEU and chrF++ use SacreBLEU's native multi-reference mode. Sentence-level metrics take the maximum score over references. Checkpoints are BERTScore with bert-base-uncased, BLEURT-20, and BARTScore with bart-large-cnn. METEOR comes from NLTK and ROUGE-L from rouge-score.

LLM-judge evaluation uses BinEval, which splits each criterion into atomic yes/no questions, with two judges, plus GPTScore. The judges were validated against 220 human-labelled outputs: two systems on the same 110 graphs, with adjudicated error types. The judges reproduce human omission labels triple by triple. Judge scores correlate only weakly with reference-based metrics, while the reference-based families agree with each other. Contradiction and omission errors grow faster with graph size for the fine-tuned models.

Limits

The fetched text stops partway through the evaluation protocol. The per-model, per-dataset score tables, the few-shot k-value results and the retrieval-strategy comparison (Section 6.1) are not in it. The paper does not give the size of the 3-point BLEU gap for any specific model.

The GPT-5 architecture is undisclosed. Prompted results reflect thinking turned off, so reasoning modes were not evaluated. Datasets were chosen for low graph-text alignment noise, which leaves noisier corpora such as GenWiki and TEKGEN untested. The judges' blindness to unsupported elaboration means BinEval scores alone cannot certify prompted outputs as faithful.