EvE: An Alternate Optimizer to Adam

What it claims

EvE is a hybrid optimizer meant for ranking or pruning configurations in hyperparameter and architecture search, not for final training. The authors present it as "a fast, gradient-aware proxy" and say it does not replace Adam as a final-stage trainer. The speedups come with these quality costs against Adam at matched budget:

• MNIST MLP: about 1 accuracy point lower.
• LoRA fine-tuning (two language datasets): roughly 9–11% higher relative test loss.
• GSM8K full test set: about 5 accuracy points lower. Fine-tuning with either optimizer lowered accuracy below the base model.

How it works

The population is fixed at four. One slot is challenged per iteration in round-robin order, so one generation is four iterations.

flowchart TD
  A[Draw minibatch B_t, re-score slot i] --> B[DE child: rand-to-pbest/1]
  B --> C{PM enabled?}
  C -- yes --> D[Mutant kept only if better]
  C -- no --> E{child beats slot i?}
  D --> E
  E -- yes --> G[Replace slot i]
  E -- no --> F[n_adam Adam steps on child, slot i's persistent moments]
  F --> H{better than child and slot i?}
  H -- yes --> G
  H -- no --> I[Slot unchanged]

• Search space: all search runs in a unit cube rescaled from the box bounds.
• Initialisation: "proximity" initialisation places one uniform anchor and three Gaussian jitters around it (σ=0.1). This makes the DE difference vectors carry local information from the start. It is the default for neural networks. The synthetic benchmarks use Latin Hypercube Sampling instead.
• DE step: z_a + F_t(z_b − z_c) + F(z_best − z_a) with F=0.5. Only the difference term gets a tiny jitter (γ=1e-4).
• Polynomial mutation: η=20, one gene per individual, gated so it can only help. It is used only on the synthetic benchmarks. It is off for networks because a single-coordinate nudge almost never wins at millions of parameters, so it is pure cost.
• Adam fallback: runs in real (unscaled) coordinates. It uses PyTorch-stock Adam with L2 weight decay folded into the gradient, not AdamW, and clamps results back to the bounds. Each slot keeps its own (m, v, t) across the run, so its fallback bursts form one continuous trajectory. The authors say any local gradient optimizer could fill this slot, but they tested only Adam.
• Scoring: every evaluation in an iteration, including the whole Adam burst, is scored on the same minibatch (common random numbers).
• Reproducibility: two generators seeded once from a single integer make runs exactly reproducible.

Evidence

Budget accounting. Budgets are counted in function evaluations. Each backward pass is converted to evaluation units using its measured wall time, not an assumed multiplier. The budgets B were:

Synthetic benchmarks.

• Problems: Ackley, Griewank, Rastrigin, Rosenbrock, Schwefel, Sphere (recentred), Zakharov.
• Dimensions: n from 2 to 10⁶, 21 seeds per cell.
• Statistics: one-sided Wilcoxon signed-rank tests, paired by seed, with Holm correction.
• The fetched text cuts off before the per-cell table, so only the 76% wins-or-ties aggregate is available.

Cost.

• Per-iteration compute is O(n_adam·C_f) in the worst case, where the fallback fires every iteration. That is the same order as n_adam plain Adam steps.
• Memory is about 13× the live parameter count, against Adam's 4×. This multiplier is constant in dimension. Appendix A reports a measured correction to this count, which is not in the fetched text.

Limits

• Learning-rate selection on test data. Where a method is reported at its best learning rate, that rate is chosen by the mean held-out test metric. Appendix A.5 covers validation-based selection, but those results are not in the fetched text.
• Monotone guarantee is deterministic-only. The best-so-far value is guaranteed non-increasing only for deterministic objectives. With minibatches, a candidate is only guaranteed to beat the incumbent on the same batch, so the training curves are not literally monotone.
• GSM8K is not like-for-like. It used a 3× larger budget than the other LoRA task.
• Base model unnamed. The 1.5B LLM is not identified in the fetched text.
• No release information. The fetched text mentions no code, package or licence.
• Truncated page. The text ends at Section 5.2. The real-task, successive-halving and ablation details (Appendices B and F) were not available.