Visual TL;DR
MoE Optimizer MemoryDriver
AdamW requires 50.6 GB for moments, dwarfing 12.6 GB bfloat16 weights
From the article 3 mentionsThe memory overhead of optimizers, particularly for large Mixture-of-Experts (MoE) models, presents a significant bottleneck.
Memory BottleneckDriver
From the article 3 mentionsThe memory overhead of optimizers, particularly for large Mixture-of-Experts (MoE) models, presents a significant bottleneck.
SkewAdam OptimizerCore
recognizes distinct parameter populations within MoE models for tailored state management
From the article 6 mentionsThe SkewAdam optimizer tackles this challenge by recognizing that the distinct parameter populations within MoE models, dense backbone, experts, and router, exhibit differing size and gradient statistics.
Differentiated State ManagementContext
tailored state management for backbone, experts, and router based on gradient statistics
From the articleInstead of a uniform approach, SkewAdam applies tailored state management: float32 momentum with a factored second moment for the backbone (5% of parameters), a factored second moment alone for the experts (95%), and an exact second moment for the router (The benefits of SkewAdam extend beyond memory efficiency.
Memory Footprint ReductionEffect
drastically cuts MoE training memory by tailoring optimizer state to parameter populations
Superior PerplexityOutcome
From the articleIn a controlled comparison over 82 million tokens, SkewAdam achieved a validation perplexity of 108.4, outperforming AdamW (126.8), Muon (120.2), and Lion (393.7).
Accessible Hardware TrainingEffect
enables training on accessible hardware, expanding research and development