Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD?

What it claims

Under DP-SGD, the authors find that untying GPT2's input embedding from its output LM head improves accuracy by up to 4.74 points on SST-2, QNLI and QQP. Untying also cuts memory by over 60%, because it lets ghost clipping run as designed. The paper's own argument goes further than the summary. Ghost clipping computes a per-example gradient norm for each layer and sums them. That sum is only exact when the parameter blocks are disjoint. With a tied matrix, the summed norm drops a cross-term. The clipping coefficients are then computed from the wrong norm, which distorts the clipping and changes the effective signal-to-noise ratio. The authors frame this as a mismatch with ghost clipping's assumptions, not an implementation bug.

How it works

• Ghost clipping, per linear layer. For sample i the gradient is the outer product δᵢaᵢᵀ. Its squared norm is ‖δᵢ‖²‖aᵢ‖², computed without ever building the gradient. The layer norms are summed, each loss is reweighted by min(1, C/‖gᵢ‖), and a second backward pass produces the clipped aggregate.
• What tying breaks. With W_out = E, the shared matrix gets gradient from both the input path and the output path. Its squared norm is ‖g_in‖² + ‖g_out‖² + 2⟨g_in, g_out⟩. The standard ghost computation omits the inner-product term.
• The exact fix, and its cost. The authors derive the corrected norm, which adds the cross-term back. They state that no known method evaluates ⟨g_in, g_out⟩ without materializing those gradients. That brings back much of the memory and compute that ghost clipping was meant to remove.
• Choice facing the engineer:

flowchart TD
  A[Tied embedding matrix] --> B{Norm computation}
  B -->|standard ghost| C[Misses cross-term: wrong clipping]
  B -->|exact norm| D[Materialize gradients: memory cost returns]
  E[Untied embeddings] --> F[Blocks separable: ghost clipping exact and cheap]

Evidence

• Models: DistilGPT2 (about 82M parameters) and GPT2 (about 124M). Each was tested as the stock weight-tied (WT) model and as an untied variant (No-WT) with separate input and output tensors.
• Training regimes compared: non-private SGD, DP-SGD with standard per-example clipping, and DP-SGD with ghost clipping.
• Headline numbers: up to 4.74 accuracy points gained from untying, and over 60% lower memory. The fetched text gives no per-task breakdown, privacy budget (ε, δ), or clipping threshold for these numbers.

Limits

• The fetched text stops partway through the experimental setup. The results tables, hyperparameters, privacy budgets and the authors' limitations section (Section VI) were not available. "Up to 4.74" is a maximum across settings, not a typical gain.
• Only small GPT-2-family models were tested, and only on GLUE classification fine-tuning. Larger models and generative tasks are not covered.
• Untying adds a second V×d matrix to train. The text does not quantify that parameter cost against the memory saved by ghost clipping.
• Before relying on the 60% memory figure, check which tied-model clipping path it is measured against: exact norm with materialized gradients, or standard per-example clipping.