Hyperparameter Scaling Laws Across MoE Sparsity Predicts Learning Rate and Batch Size to 1/64 Sparsity

Hyperparameter Scaling Laws Across MoE Sparsity shows conventional scaling breaks when you push sparsity to 1/64, and the activation ratio itself predicts where the optima move.

The gap is not subtle.

At 80M activated parameters and 20B training tokens, the dense model and the A=1/64 MoE prefer different learning rates and batch sizes even when activated count, total count, and compute are matched. Neither total parameters nor activated parameters explains it.

To map the shift, the authors ran 1,800 pre-training runs across six activated-parameter scales from about 10M to 324M, up to 6B total non-embedding parameters, processing about 20 trillion tokens at 200,000 equivalent H800 GPU-hours.

Two regimes reconcile prior conflicts. At fixed sparsity, optimal batch size follows a power law in training tokens D, while optimal learning rate follows a power law in training compute C=MD and stays stable across different M/D allocations at the same C. Across sparsity, the activation ratio A = E_act / E_tot enters as a multiplicative power-law correction to both.

That gives a unified family h*(X,A)=k_h X^{gamma_h} A^{delta_h}, with (h,X) in {(eta,C),(B,D)}. Specifically eta*(C,A)=k_eta C^{gamma_eta} A^{delta_eta} and B*(D,A)=k_B D^{gamma_B} A^{delta_B} where delta_eta >0 and delta_B <0.

Fitted on A in {1,1/4,1/16,1/32}, the laws are eta with k=0.8343, gamma=-0.1385, delta=0.1361 and B with k=6.4765, gamma=0.5181, delta=-0.0841, with C in non-embedding FLOPs and D,B in tokens. Decreasing A lowers the optimal learning rate and raises the optimal batch size.

Fit is not just descriptive. Under grouped leave-one-activation-ratio-out and leave-one-active-scale-out, the multiplicative form beats scale-only and additive families, and ties the more flexible log-interaction variant while using one fewer parameter.

Joint extrapolation is tested on a held-out 12B total parameter MoE with only 1/64 experts activated, N=324M, D=159B, C=3e20 FLOPs, beyond the fitting ranges in A, D, and C. The frozen prediction at 6.92e-4 learning rate and 5.84e6 batch tokens sits on the near-optimal plateau, with refitted DeepSeek and Step laws trailing by 1.37 and 1.03 per mille training loss gaps.

Controls isolate what A means. Holding A=1/32 while doubling active and total experts and halving expert width leaves optima unchanged, so the relation transfers across granularity. Holding total experts and capacity fixed while moving A from 1/32 to 1/16 shifts both optima downward as predicted. The intuition offered is gradient noise per expert scales as A*B, so sparser models want larger global batches but not enough to keep A*B* constant, so learning rate still falls.

That sparsity correction is the missing term earlier joint laws did not model. Recent work on routed language models has derived joint scaling laws that unify dense and MoE performance by adding active parameters and dataset size, but they treat sparsity largely through expert count rather than a continuous activation ratio factor.

The scope is narrow by design. Evidence comes from one hybrid linear-attention/MLA backbone, one data mixture, Muon optimizer, and sigmoid auxiliary-loss-free routing, with no multi-seed repeats due to cost and only one ultra-sparse target for joint extrapolation. Intervals for functional-form comparison overlap, so the data do not rule out exponent interaction with A.

The prescription turns sparsity from architecture trivia into a predictor you can fit: use C for learning rate, D for batch size, multiply by A.