Cursor Open-Sources MoE Megakernel

Cursor has open-sourced its proprietary Mixture-of-Kittens (MoK) megakernel, a specialized component designed to accelerate Mixture-of-Experts (MoE) model training on NVIDIA's NVL72 hardware. This move aims to address a significant bottleneck in training large AI models, particularly agentic ones like Cursor's own Composer.

The Mixture-of-Kittens project, detailed by the Cursor team, is more than just an optimization; it's a ground-up redesign of the MoE layer. MoE models, which route different parts of an input to specialized 'expert' sub-networks, are powerful but computationally intensive. The communication overhead between GPUs, sending token data to the correct experts and gathering results, often becomes the limiting factor, especially as models scale to thousands of GPUs.

From Optimization to Integration

Cursor's previous efforts focused on optimizing specific parts of the MoE layer, like compute kernels. However, they found that inter-GPU communication was the real choke point. MoK tackles this by integrating all MoE communication and computation into a single, deterministic kernel. This approach is particularly suited for the NVL72 platform, a multi-node server designed for high-bandwidth, fine-grained communication across its 72 GPUs via NVLink. The integrated Grace CPUs in these systems can also become a bottleneck if not managed carefully; MoK minimizes CPU involvement to keep GPUs busy.

Strategic Communication Choices

A key innovation in MoK lies in its approach to inter-GPU communication. Instead of relying solely on push-based mechanisms common in other MoE implementations, MoK strategically chooses between push and pull communication for different stages of the process. This decision is driven by maximizing NVLink bandwidth utilization and minimizing signaling overhead. While push-based methods can saturate links with less protocol overhead, pull-based transfers, surprisingly, can achieve better practical bandwidth on NVLink for MoE workloads due to how data is split and balanced across bidirectional lanes. For example, MoK uses pull-based dispatch for forward passes and push-based combine, then reverses these for the backward pass. This strategy eliminates costly cross-GPU signaling, allowing computation to begin immediately upon data arrival.

Performance Gains

The results are substantial. Cursor reports that MoK delivers up to 2.37x higher MXFP8 forward throughput compared to public baselines on GB300 NVL72s. More importantly, in their production training stack, MoK increased end-to-end tokens per second by 1.41x. This level of improvement can significantly reduce the time and cost associated with training state-of-the-art large language models, especially those employing the MoE architecture.

Why This Matters for the AI Industry

The open-sourcing of MoK is a significant development for the AI community. MoE architectures are becoming increasingly prevalent in large models due to their efficiency and scalability. By providing a high-performance, open-source kernel optimized for specific, powerful hardware like the NVL72, Cursor is lowering the barrier to entry for researchers and developers working with these models. This could accelerate innovation in areas like agentic AI, where models need to perform complex reasoning and task execution. Companies that have adopted similar MoE architectures, such as Mistral AI or others developing large open-weight models, may find MoK a valuable tool to boost their training efficiency. For hardware vendors, this highlights the ongoing need for software that can fully exploit the capabilities of advanced interconnects like NVLink.

StartupHub Angle

For founders building the next generation of AI infrastructure, MoK exemplifies a critical trend: deep hardware-software co-design. Just as early cloud startups focused on optimizing for specific AWS instances, today's AI infrastructure companies must engineer solutions tailored to the unique architectures of NVIDIA's latest accelerators and interconnects. This isn't just about raw compute; it's about orchestrating communication with extreme precision. The MoK release suggests that for startups aiming to compete in the LLM training space, specialized kernels that unlock hardware potential, rather than relying on generic frameworks, will be key differentiators.