Technology

Faster LLMs by Reshaping Sparsity

Sakana AI and NVIDIA unveil a new method that reshapes sparsity in LLMs to boost GPU efficiency, achieving over 20% speedups.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Abstract visualization of data sparsity and GPU architecture interaction.
A visual representation of how sparse data is optimized for GPU processing.Credit: Sakana

Making large language models (LLMs) faster and lighter often runs into a hardware paradox: doing less computation can paradoxically slow them down. This stems from GPUs being optimized for dense, predictable data blocks, while the natural sparsity in LLMs creates irregular memory access.

A new collaboration between Sakana AI and NVIDIA aims to resolve this mismatch. Instead of forcing LLMs to adapt to GPU limitations, their approach reshapes the sparsity to fit the hardware. This work introduces novel open-source GPU kernels and data formats for optimizing sparse transformer language models, detailed in a recent paper.

The core of the innovation is a new format called TwELL (Tile-wise ELLPACK). This hybrid representation dynamically routes the 99% of sparse tokens through a fast path, using a dense backup matrix for less frequent, intensive tokens. This design ensures GPUs can process data more efficiently.

The contribution is twofold: TwELL, a sparse packing format designed for optimized tiled matrix multiplication kernels, and custom CUDA kernels that fuse multiple sparse operations. These kernels compress TwELL to minimize activation sizes, maximizing throughput.

Benchmarks at billion-parameter scale demonstrated over 20% speed improvements and significant reductions in peak memory and energy consumption. This research offers a promising path toward more efficient LLMs, impacting everything from optimizing LLM inference and training to enabling smaller, more accessible models, as discussed in perspectives on optimizing LLM inference and training.

Mentioned in this story

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsSparse AITransformer ModelsLLM OptimizationGPU ComputingNvidiaSakana AIICML 2026CUDA Kernels

StartupHub.ai Staff

Written by

StartupHub.ai StaffEditorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Reviewed by

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.