LookAtMyQuant is a quantization compiler designed to compress large language models (LLMs). It operates by measuring the impact of compression on each model piece and then applies a code to fit a specified size budget with minimal performance degradation. The compiler's methodology and results are publicly available.
LookAtMyQuant 2 compressed the Qwen3.5-4B model to 1.23 GB, achieving 2.3 bits per weight. This compressed version solved 330 out of 540 exact-answer math problems, compared to 328 for Prism ML's Bonsai 2 Ternary 4B at 1.20 GB and 359 for the original model. Its KL divergence from the original on held-out text was 0.45 nats, better than Bonsai's 0.59.
The compiler also reduced Qwen3.8-27B from 53.7 GB to 7.2 GB, at 2.1 bits per weight. This compressed model solved 401 of 540 math problems, matching the original model's performance. It also scored higher than Unsloth's UD-IQ2_XXS GGUF of the same size on HumanEval (73.8% vs 67.1%), MMLU (78.8% vs 75.0%), and GSM8K (94.5% vs 92.7%). A 4-bit DSpine drafter (1.02 GB) for Qwen3.8-27B was also created, which approximately doubles decode speed on Apple silicon for math and code tasks.
The same compiler, combined with proximal fine-tuning (SQYFR) and extended to a 3x-looped MoE (26.4B physical / 5.4B active), compressed thinkingmachines/Inkling-2-Mini. The text weights were reduced from 18.89 GB to 5.38 GB (1.63 bits/weight) and 8.87 GB (2.69 bits/weight). This compression utilized 4 KB SRAM hashed block-vector trellis codes (ht2/ht3), row trellis codes (rtr1/rtr4), and a tr2 floor on the tail lm_head bucket (head.7).
The compression recipe involves post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10x, and bits allocated per row by Fisher information. This is followed by 4,000 steps of distillation, where each layer is re-quantized from the original weights. The code for this recipe is available under the MIT license, and the compressed files, such as lm2-qwen35-4b-1.2gb, are packed for the Apple Silicon runtime.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
LookAtMyQuant, a quantization compiler, has demonstrated significant compression of large language models (LLMs) while maintaining performance. It reduced Qwen3.5-4B to 1.23 GB and Qwen3.8-27B to 7.2 GB, outperforming other quantization methods in benchmarks. The compiler also compressed Inkling-2-Mini from 18.89 GB to 5.38 GB, using specialized trellis codes and distillation.