HeuFouFT: Task-Guided Metaheuristic Coordinate Search for Fourier Fine-Tuning

What it claims

On GPT-2-Medium / E2E, choosing FourierFT's frequency coordinates by task-driven search beats fixed sampling. With 624 coefficients per adapted matrix, versus 1,000 for FourierFT and LoCA, PSO-searched coordinates reach BLEU 70.9 against LoCA's 70.3. The selection is not free: the full pipeline takes about 12 hours on one A100 80GB, paid once per task.

How it works

FourierFT builds ΔW = α·Re(IDFT(sparse spectrum)). The coordinate set E is shared across layers, and each layer trains its own n real coefficients. HeuFouFT searches for E on a 1024×1024 grid:

1. Block probes. The spectrum is split into a 10×10 grid of blocks. Each block is fine-tuned on 10% of the training data for one epoch and scored on validation. Scores combine into a heuristic intensity: HI = 0.3 BLEU + 0.2 METEOR + 0.2 ROUGE-L + 0.15 (NIST + CIDEr), using normalized metrics. The resulting map correlates with LoRA ΔW spectral energy at r = 0.85.
2. Metaheuristic search starts from that map, with three variants: - GA-SA: two-point crossover, mutation rate 0.05, SA temperature starting at 1000 with ×0.95 decay. - PSO: inertia 0.9→0.4. Each coordinate takes a sigmoid-gated ±1 step modulo d. - Cuckoo Search: Lévy flights with λ = 1.5 and step size 0.5→0.1.
3. Surrogate filter. A Random Forest scores every new population, and only the top 30% are proxy-fine-tuned to get their HI fitness. The surrogate reaches R² = 0.86 on held-out candidates.
4. Final fine-tune uses the fixed E.

flowchart TD
  A[10x10 block probes] --> B[Intensity map]
  B --> C[Init population]
  C --> D[Random Forest filter]
  D -->|bottom 70%| X[Discard untrained]
  D -->|top 30%| E[Proxy fine-tune, val HI]
  E --> F{Converged?}
  F -->|no| C
  F -->|yes| G[Fixed E, final fine-tune]

Evidence

Results on the E2E test set (42k train, 4.6k val, 4.6k test), mean over 3 seeds. Only Q and V projections are adapted, with AdamW, batch size 4 and 3 epochs.

• Ablation: dropping simulated annealing (vanilla GA) lowers all five metrics, for example BLEU 69.6 vs 70.6.
• Gaussian band-pass sweep: f_c from 0 to 500 with W = 20 gives a different best f_c for each metric.
• Different coordinates, similar scores: PSO puts 84.5% of its coordinates in the middle band, while CS puts 79.6% in the high band. The authors conclude the gain comes from task guidance rather than from the choice of optimizer.

How to use it

Code: https://github.com/pierrewang2024-collab/HeuFouFT. The surrogate's features, labels and hyperparameters are documented only in the repository, not in the paper.

Search cost on one A100 80GB in FP16:

• Intensity map: 1.25 h.
• Search: GA-SA 2.92 h (175 iterations), PSO 3.00 h (240 iterations), CS 1.26 h (151 iterations).
• Full pipeline: about 12 h.

Cost after selection, as a share of full fine-tuning:

CS is the cheapest search; PSO gives the best scores.

Limits

• Only one model (GPT-2-Medium) and one task (E2E) are tested. Nothing is shown for larger LLMs, other tasks, or vision.
• E is fixed across the three seeds, so the reported standard deviations exclude search variance.
• Some margins over LoCA are within noise. PSO's ROUGE-L ties LoCA at 68.5, and the BLEU gap is 0.6 with std ±0.1–0.2.
• The parameter and FLOP comparisons leave out the roughly 12 h search. That cost only amortizes if E is reused, and the paper does not test reuse across tasks.
• The HI weights are a hand-set preference over metrics. Search fitness, probes and surrogate labels all come from the validation split.