# SkewAdam: Rethinking MoE Optimizer Memory
_SkewAdam drastically cuts MoE training memory by tailoring optimizer state to parameter populations, achieving superior perplexity and enabling training on accessible hardware._
**Published:** 2026-07-22
**Source:** https://www.startuphub.ai/ai-news/ai-research/2026/skewadam-rethinking-moe-optimizer-memory
---
The memory overhead of optimizers, particularly for large [Mixture-of-Experts](/ai-news/technology/2026/cursor-s-warp-decode-boosts-moe-inference) (MoE) models, presents a significant bottleneck. For a 6.78B-parameter MoE, AdamW requires 50.6 GB for first and second moments, dwarfing the 12.6 GB of bfloat16 weights. This disparity highlights a critical area for optimization.

MoE Optimizer MemoryDriver

AdamW requires 50.6 GB for moments, dwarfing 12.6 GB bfloat16 weightsFrom the article 3 mentionsThe memory overhead of optimizers, particularly for large Mixture-of-Experts (MoE) models, presents a significant bottleneck.createsMemory BottleneckDriverFrom the article 3 mentionsThe memory overhead of optimizers, particularly for large Mixture-of-Experts (MoE) models, presents a significant bottleneck.addressesSkewAdam OptimizerCorerecognizes distinct parameter populations within MoE models for tailored state managementFrom the article 6 mentionsThe SkewAdam optimizer tackles this challenge by recognizing that the distinct parameter populations within MoE models, dense backbone, experts, and router, exhibit differing size and gradient statistics.usesDifferentiated State ManagementContexttailored state management for backbone, experts, and router based on gradient statisticsFrom the articleInstead of a uniform approach, SkewAdam applies tailored state management: float32 momentum with a factored second moment for the backbone (5% of parameters), a factored second moment alone for the experts (95%), and an exact second moment for the router (The benefits of SkewAdam extend beyond memory efficiency.leads toMemory Footprint ReductionEffectdrastically cuts MoE training memory by tailoring optimizer state to parameter populationsSuperior PerplexityOutcomeFrom the articleIn a controlled comparison over 82 million tokens, SkewAdam achieved a validation perplexity of 108.4, outperforming AdamW (126.8), Muon (120.2), and Lion (393.7).Accessible Hardware TrainingEffectenables training on accessible hardware, expanding research and development

## Memory Footprint Reduction via Differentiated State Management

The [SkewAdam optimizer](https://arxiv.org/abs/2607.19058v1) tackles this challenge by recognizing that the distinct parameter populations within MoE models, dense backbone, experts, and router, exhibit differing size and gradient statistics. Instead of a uniform approach, SkewAdam applies tailored state management: float32 momentum with a factored second moment for the backbone (5% of parameters), a factored second moment alone for the experts (95%), and an exact second moment for the router (<0.01%). This strategy slashes the total optimizer state to a mere 1.29 GB, a mere 2.6% of AdamW's requirement. Consequently, peak training memory plummets from 81.4 GB to 31.3 GB, fitting comfortably within a 40 GB accelerator budget.

## Performance Gains Beyond Memory Savings

The benefits of SkewAdam extend beyond memory efficiency. In a controlled comparison over 82 million tokens, SkewAdam achieved a validation perplexity of 108.4, outperforming AdamW (126.8), Muon (120.2), and Lion (393.7). Crucially, this performance was attained without sacrificing router load balance, which settled within 1% of its uniform floor. Further analysis indicates that the accuracy gains stem from retaining momentum, a benefit shared across uniform optimizers, while the tiered allocation primarily yields memory savings at no accuracy cost. Even tuned baselines like AdamW (118.5) and Adafactor (139.7) could not close the performance gap, underscoring the efficacy of SkewAdam's approach.
---
Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.