Task Format as a Factor in Adaptation: A Controlled Comparison of LoRA and Full Fine-Tuning for Prompt-Safety Classification

What it claims

On a matched pair of 400M Ettin models (encoder and decoder) trained for prompt-safety classification, the choice between LoRA and full fine-tuning matters less than the learning rate and the training target. Whether the label comes from a classification head or a generated verdict token makes little difference under either regime. With pre-specified learning rates, full fine-tuning leads LoRA by 2 to 3 pp of in-domain balanced accuracy. On the encoder, the two regimes are equivalent once both learning rates come from the same search, within an equivalence margin fixed before the runs. On the content-safety corpora, the head-based classifiers also rank prompts worse than the prompted 8B judge. The authors conclude that a new decision threshold would not close that gap.

How it works

The three task formats differ in where the loss is computed:

1. Classification head. Two-class cross-entropy through a projection trained from fresh weights. The backbone only needs to make its features separable.
2. Generated verdict token. Next-token loss on one token through the pretrained LM head. The weights must move probability mass across about 50,000 vocabulary tokens onto the verdict word.
3. Verdict plus reason. The verdict is followed by a one-sentence reason. Because the label is generated first, the reason can affect the label only through training.

The Ettin encoder and decoder share one pretraining mixture, recipe and tokenizer. They differ only in objective and attention mask, so architecture is the only pretraining difference. This removes the scale and architecture confound in earlier studies, such as Li et al. (2023), which compared 3B+ decoders against 110M to 355M encoders.

For the reason format, the paper separates the sources of the gain. A content-free sentence of the same length, placed after the label, tests whether length alone explains it. Reasons borrowed from another item with the same label, or with any label, test whether content and agreement with the label matter. Earlier controls of this kind (Wadhwa et al., 2024) put the filler before the label.

Evidence

• Training data is a single corpus of 29K rows.
• Each trained cell has 15 seeds. The two reason cells have 9.
• Regime equivalence is tested against a margin fixed in advance, using a 90% confidence interval as described by Lakens (2017).
• The shared learning-rate search was run for the encoder only.

The reason target adds 4.0 pp to LoRA and closes its gap with full fine-tuning. The length-matched content-free sentence adds nothing.

Out of distribution, every trained 400M model trails a prompted 8B judge by 8 to 23 pp on synthetic adversarial jailbreaks. The trained models flag 27 to 66% of benign but harmful-looking prompts as unsafe. The 8B flags 2 to 3%. The trained models fall behind on two of the three out-of-distribution corpora.

For context, the paper cites Chen et al. (2022). Under a controlled protocol with 10 to 20 seeds, LoRA trailed full fine-tuning by 0.7 to 1.9 pp on tasks with more than 1,000 training examples.

How to use it

The authors say they will release the harness, the frozen analysis plans and per-run records. As of the October 2026 preprint, these were submitted as a zipped archive for review, and the public release is pending. The page gives no repository link, commands or licence.

Limits

• The study covers prompt-safety classification only, at 400M parameters, on one 29K-row training corpus.
• The finding that tuned learning rates make the regimes equivalent holds for the encoder only. The decoder was not given the same learning-rate search.
• The fetched text stops at the start of Section 3. LoRA rank, target layers, scaling, the identity of the 8B judge and the out-of-distribution corpus names are not available here.