Crusoe Keeps 4-Bit Weights to Cut LoRA Memory by 75%

Crusoe Blog says its post-training team built LoRA training that keeps frozen base weights in 4-bit and dequantizes them inside the kernel.

LoRA, or Low-Rank Adaptation, freezes the base model and trains two small matrices that add about one percent of parameters.

A 1 trillion parameter model like Kimi-K2.6 needs at least 2 terabytes for base weights in bf16, or four HGX H100 nodes, according to Crusoe.

In unmerged LoRA inference Crusoe keeps the base path in mxfp4 with activations quantized to mxfp8 while the adapter path stays in bf16.

On Blackwell the matmul runs natively in mxfp4, but on Hopper Crusoe still stores 4-bit and dequantizes to bf16 inside the kernel.

In the backward pass Crusoe needs only gradients for A, B and the input, so frozen W never needs a bf16 copy or optimizer update.

DeepSeek V4 also cuts single-token FLOPs and KV cache to about ten percent of DeepSeek V3.2 at one million tokens, per Codersera's setup guide.

Crusoe reports no accuracy drop from the train-inference precision mismatch, but the result relies on the quantization passthrough assumption used in quantization-aware training.

Serverless Fine-Tuning with this dequant-on-the-fly kernel is now generally available in Crusoe Intelligence Foundry and bills per training token.