CPU LLMs: Architecture First, Size Later

The conventional wisdom for developing small language models (SLMs) has been to train them like their larger counterparts and then attempt to compress them for CPU deployment. This approach often results in a compromise between capability and efficiency. However, a new research effort from Christos Koutsiaris challenges this paradigm by adopting an inverse strategy: fixing the target hardware and user experience from the outset.

Architecting for the CPU, Token by Token

Instead of squeezing large models onto CPUs, this work began by defining the target environment: a standard CPU capable of processing one token at a time with 4-bit weights. The architecture was then specifically chosen to suit these constraints. The resulting model features a hybrid design, retaining full attention mechanisms in only 6 of its 18 blocks. The remaining 12 blocks utilize short convolutions with a memory footprint limited to two timesteps, meaning they do not re-read an ever-growing cache regardless of conversation length. This deliberate architectural choice is key to its efficiency.

The model was trained from scratch on 59.9 billion tokens. On a five-task benchmark, it achieved a score of 47.31, surpassing a pre-defined bar of 42.20. This performance is remarkable given that it outperforms models like GPT-2 124M, Pythia-160M, OPT-125M, and GPT-neo-125M, all of which were trained on three to six times more data. Furthermore, it exceeds the published score of MobileLLM-125M, a model trained on a trillion tokens. The validation bits-per-byte stands at an efficient 0.8685.

Validation Beyond Training Data

To isolate the impact of the novel architecture from the training methodology, the researchers trained a conventional all-attention model of the same size using the identical dataset. Crucially, the winning metric was established before training commenced. The hybrid architecture not only won this chosen quality metric by 0.81% but also matched its performance on downstream tasks. It produced a 4-bit file that was 6.3% smaller and decoded 1.76 times faster at 2048 tokens of context. Against an external model of similar size, it achieved a 2.08x speedup.

This speed advantage, which is negligible at empty contexts and grows with context length, aligns perfectly with the architectural predictions. This behavior is distinct from what would be expected from a merely leaner model. A simple bandwidth calculation predicts only a 1.17x speedup, indicating that memory volume alone does not account for the observed performance gap. The researchers also detailed what did not yield positive results, including an unmitigated 4-bit quality cost, a significant portion of convolution channels proving inert, and a vocabulary size deemed too large for the model's scale.