SkewAdam: Rethinking MoE Optimizer Memory

SkewAdam drastically cuts MoE training memory by tailoring optimizer state to parameter populations, achieving superior perplexity and enabling training on accessible hardware.

Jul 22 at 8:01 PM5 min read

Diagram illustrating the memory breakdown of AdamW vs SkewAdam for MoE training. — SkewAdam significantly reduces optimizer state memory in MoE models.

Visual TL;DR. MoE Optimizer Memory creates Memory Bottleneck. Memory Bottleneck addresses SkewAdam Optimizer. SkewAdam Optimizer uses Differentiated State Management. Differentiated State Management leads to Memory Footprint Reduction. Memory Footprint Reduction enables Superior Perplexity. Memory Footprint Reduction allows Accessible Hardware Training.

MoE Optimizer Memory: AdamW requires 50.6 GB for moments, dwarfing 12.6 GB bfloat16 weights
Memory Bottleneck: significant bottleneck for large Mixture-of-Experts (MoE) model training
SkewAdam Optimizer: recognizes distinct parameter populations within MoE models for tailored state management
Differentiated State Management: tailored state management for backbone, experts, and router based on gradient statistics
Memory Footprint Reduction: drastically cuts MoE training memory by tailoring optimizer state to parameter populations
Superior Perplexity: achieved superior perplexity in controlled comparison over 82 million tokens
Accessible Hardware Training: enables training on accessible hardware, expanding research and development

Visual TL;DRQuickExplainDeeper

The memory overhead of optimizers, particularly for large Mixture-of-Experts (MoE) models, presents a significant bottleneck. For a 6.78B-parameter MoE, AdamW requires 50.6 GB for first and second moments, dwarfing the 12.6 GB of bfloat16 weights. This disparity highlights a critical area for optimization.

Memory Footprint Reduction via Differentiated State Management

The SkewAdam optimizer tackles this challenge by recognizing that the distinct parameter populations within MoE models, dense backbone, experts, and router, exhibit differing size and gradient statistics. Instead of a uniform approach, SkewAdam applies tailored state management: float32 momentum with a factored second moment for the backbone (5% of parameters), a factored second moment alone for the experts (95%), and an exact second moment for the router (<0.01%). This strategy slashes the total optimizer state to a mere 1.29 GB, a mere 2.6% of AdamW's requirement. Consequently, peak training memory plummets from 81.4 GB to 31.3 GB, fitting comfortably within a 40 GB accelerator budget.

Performance Gains Beyond Memory Savings

The benefits of SkewAdam extend beyond memory efficiency. In a controlled comparison over 82 million tokens, SkewAdam achieved a validation perplexity of 108.4, outperforming AdamW (126.8), Muon (120.2), and Lion (393.7). Crucially, this performance was attained without sacrificing router load balance, which settled within 1% of its uniform floor. Further analysis indicates that the accuracy gains stem from retaining momentum, a benefit shared across uniform optimizers, while the tiered allocation primarily yields memory savings at no accuracy cost. Even tuned baselines like AdamW (118.5) and Adafactor (139.7) could not close the performance gap, underscoring the efficacy of SkewAdam's approach.

© 2026 StartupHub.ai. All rights reserved. Do not enter, scrape, copy, reproduce, or republish this article in whole or in part. Use as input to AI training, fine-tuning, retrieval-augmented generation, or any machine-learning system is prohibited without written license. Substantially-similar derivative works will be pursued to the fullest extent of applicable copyright, database, and computer-misuse laws. See our terms.

#AI Research #Optimization #MoE #Deep Learning #Hardware Acceleration