Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training

What it claims

Concept augmentation adds training sentences in which one word is swapped for a synonym or hypernym. The paper tests it on nouns, verbs, adjectives and all three together while post-training Llama-3-8B. The post-training corpus matters most for perplexity: augmentation lowers perplexity by 5.5% on average on news but raises it by 8.2% on arXiv. Verbs give the best mean downstream gain in six of eight relation × corpus settings. Augmenting all POS at once does not reliably beat the best single POS. Perplexity gains predict downstream gains only weakly.

How it works

The method follows the pipeline of Iyer et al. (2026) and keeps the standard next-token prediction loss. Only the data changes.

1. Corpora: arXiv abstracts, news text, YouTube comments, and a combined corpus, downsampled to balance volume across variants.
2. Target POS: nouns, verbs, adjectives, or all three.
3. Relation: synonyms (coach → trainer) or hypernyms (coach → professional).
4. Concept source: context-aware alternatives generated by an LLM from the full sentence, or dictionary alternatives from WordNet with no context.
5. Baseline: each augmented model has a size-matched vanilla model. It repeats the original sentences to match example count, so the comparison isolates substitution from training volume.

The study produces 96 post-trained models. Each is evaluated with LoRA sequence classification on seven tasks: GLUE, SNLI, Logical Fallacy, Spam, Hate Speech, Emotion and one more not named in the fetched text. Checkpoints are chosen by lowest validation loss. This replaces the instruction-style fine-tuning used by Iyer et al.

Evidence

All numbers are relative to the matched vanilla model.

• Perplexity by corpus: news +5.5%, arXiv −8.2%, YouTube −2.6%, combined −2.5% (positive means lower perplexity).
• Perplexity by POS: On arXiv, only all-POS augmentation helps on average, and verbs degrade less than nouns or adjectives. On news, hypernym adjectives improve 14.4% and hypernym verbs 13.8%. On YouTube, verb synonyms improve 12.6%. On the combined corpus, adjective hypernyms improve 9.8% while adjective synonyms worsen perplexity substantially.
• Downstream accuracy, verbs best: hypernym arXiv +4.5, hypernym news +2.9, hypernym combined +0.5, synonym arXiv +2.2, synonym YouTube +2.8, synonym combined +0.9. The exceptions are hypernym YouTube, where nouns win, and synonym news, where adjectives win (+3.4).
• Task sensitivity: GLUE swings most, for example +30.9 for hypernym verbs on arXiv and +28.8 for synonym adjectives on news. GLUE, SNLI and Logical Fallacy are much more sensitive than Spam, Hate Speech and Emotion.
• Perplexity vs downstream: the correlation is r = 0.38 (p = 0.033) for context-aware and r = 0.30 (p = 0.096) for dictionary augmentation. Models fall in all four quadrants, so the authors treat downstream accuracy as the primary criterion.
• Reproduction: noun-only results reproduce the broad trends of Iyer et al. (2026). Details are in Appendix A.

Limits

• Results come from one base model, Llama-3-8B, and classification tasks only.
• The best POS changes by domain, relation and task, so no single setting transfers across conditions.
• Augmentation can hurt: on arXiv, most single-POS conditions raise perplexity.
• The fetched text stops partway through Section 5.4. The CKA and linear-probing results, the hyperparameters (Appendix C) and the per-condition tables (Appendix D) are not included. The page states no code, data or weights release.