AI Research

LookAtMyQuant squeezes 27B model into 7.2GB

LookAtMyQuant claims 12x lower KL than Q4_K_M and ships 7.2GB Qwen3.8-27B matching the original on math.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Quantization compiler compressing large language model weights
Credit: GitHub

Key facts

  • LookAtMyQuant claims 12x lower KL than Q4_K_M on Gemma 4 E2B
  • Qwen3.5-4B hits 1.23 GB and solves 330 of 540 math tasks
  • Qwen3.8-27B hits 7.2 GB and solves 401 of 540 math tasks
  • harness-hub lists Qwen3.8-27B-pi-adapted GGUF for llama.cpp

LookAtMyQuant says its quantization compiler cuts KL to one twelfth of Q4_K_M at the same size on Gemma 4 E2B. LookAtMyQuant says it measures damage from each compressed piece and then picks codes to fit a size budget.

That is a compiler claim, not a single checkpoint.

LookAtMyQuant says Qwen3.5-4B drops to 1.23 GB at 2.3 bits per weight and solves 330 of 540 math problems. LookAtMyQuant puts Prism ML Bonsai 2 Ternary 4B at 328 at 1.20 GB and the original at 359, with KL at 0.45 nats versus 0.59. LookAtMyQuant says the recipe uses trellis codes calibrated on worked math with digit tokens weighted 10x plus 4,000 steps of distillation for about $34 of GPU time.

LookAtMyQuant says Qwen3.8-27B drops from 53.7 GB to 7.2 GB at 2.1 bits per weight and solves 401 of 540 math problems, matching the original. LookAtMyQuant puts Unsloth UD-IQ2_XXS at 348 at the same size, with HumanEval at 73.8% against 67.1% and MMLU at 78.8% against 75.0%. The Apple Silicon build ships with a 1.02 GB DSpine drafter that LookAtMyQuant says roughly doubles decode speed on math and code.

Qwen3.8-27B at 7.2 GB meets pi adapters

Harness Hub lists harness-hub/Qwen3.8-27B-pi-adapted as a LoRA adapter trained with reinforcement learning inside pi on Qwen 27B. Harness Hub lists a GGUF version of that adapter for llama.cpp, a different bet from full weight compression for MLX. LookAtMyQuant is arguing for fewer bits everywhere, while Harness Hub is arguing for frozen base weights plus a learned adapter.

LookAtMyQuant says Inkling-2-Mini compresses from 18.89 GB to builds at 5.38 GB, 8.87 GB, 11.18 GB and 12.76 GB, with the 12.76 GB build matching the teacher at 193 of 200 on GSM8K and 152 of 164 on HumanEval. LookAtMyQuant says SmolLM2-360M needs 2,312 sensitivity measurements and about 50 minutes on an L40S to reach perplexity 16.64 and KL 0.0926 against 15.31 original. No independent reproduction is cited in the repo and most speed numbers cover Apple M5 Max with Metal kernels. Who verifies 0.45 nats and 401 of 540 outside Modal H100 runs and MLX?

Sources

TopicsLookAtMyQuantQwen3.8-27BQwen3.5-4BGemma 4 E2BUnslothPrism MLInkling-2-Mini

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.