Training instability in Adam may be more predictable than it looks. A new study mapping optimizer dynamics across the (beta1, beta2) plane finds that macroscopic loss spikes are not random, but are separated from stable training by an approximately linear boundary of the form 1 minus beta2 equals C times 1 minus beta1. The work, Beyond Quadratic Loss: The Stability Phase Diagram of Adam, tests this relationship across a range of model-task settings and shows the same simple rule governs when training stays smooth and when it spikes.
The finding reframes how practitioners should think about Adam's two momentum timescales. Rather than tuning beta1 and beta2 independently, the boundary suggests they trade off linearly, with implications for how to stay in the stable region without sacrificing adaptivity. For teams training large models where a single spike can waste days of compute, a predictable stability diagram offers a practical tuning prior and a diagnostic for instability that goes beyond quadratic loss approximations.
From stability to efficiency in distributed and adversarial training
The stability result lands alongside two complementary advances in optimization that together point toward cheaper, simpler training. Revisiting Distributed Sign-Based Variance Reduction tackles the communication bottleneck in distributed learning, showing that naive majority voting over signs introduces bias under heterogeneous data and proposing a correction that restores optimal convergence rates for both nonconvex stochastic and finite-sum optimization. In practice, that means sign-based compression can be used without paying a hidden bias cost.
Matching Multi-Loop Complexities with a Single Loop: Optimal Optimization Stationarity and Best-Known Game Stationarity in Nonconvex--Concave Minimax Optimization addresses a different cost, the complexity of minimax optimization used in adversarial training, robustness and generative modeling. The authors introduce a single-loop projected damped extragradient method with dual momentum and a moving proximal center that achieves the best-known complexity among single-loop first-order methods for both optimization and game stationarity. Together with the Adam phase diagram and the corrected sign-based reduction, the three papers suggest a converging lesson: stable training does not require more loops or more communication, but better understood dynamics and tighter algorithms.