← All stories
● Covered by 1 source · 1 reportMedium impact1 neutral

LookAtMyQuant Compiler Achieves High Compression for LLMs, Including Inkling-2-Mini

🔄 Updated 8h ago
New to BrevFeed? We gather this story from every outlet covering it into one summary — ranked by real-world impact, not just the latest headline — so you never miss what matters. What is BrevFeed? →

Key points

  • LookAtMyQuant 2 compresses Qwen3.5-4B to 1.23 GB (2.3 bits/weight).
  • Qwen3.8-27B compressed to 7.2 GB (2.1 bits/weight), matching original math problem solving.
  • Inkling-2-Mini compressed from 18.89 GB to 5.38 GB (1.63 bits/weight).
  • Method uses post-training quantization with trellis codes and distillation.

Quantization Compiler Overview

LookAtMyQuant is a quantization compiler designed to compress large language models (LLMs). It operates by measuring the impact of compression on each model piece and then applies a code to fit a specified size budget with minimal performance degradation. The compiler's methodology and results are publicly available.

Performance on Qwen Models

LookAtMyQuant 2 compressed the Qwen3.5-4B model to 1.23 GB, achieving 2.3 bits per weight. This compressed version solved 330 out of 540 exact-answer math problems, compared to 328 for Prism ML's Bonsai 2 Ternary 4B at 1.20 GB and 359 for the original model. Its KL divergence from the original on held-out text was 0.45 nats, better than Bonsai's 0.59.

The compiler also reduced Qwen3.8-27B from 53.7 GB to 7.2 GB, at 2.1 bits per weight. This compressed model solved 401 of 540 math problems, matching the original model's performance. It also scored higher than Unsloth's UD-IQ2_XXS GGUF of the same size on HumanEval (73.8% vs 67.1%), MMLU (78.8% vs 75.0%), and GSM8K (94.5% vs 92.7%). A 4-bit DSpine drafter (1.02 GB) for Qwen3.8-27B was also created, which approximately doubles decode speed on Apple silicon for math and code tasks.

Compression of Inkling-2-Mini

The same compiler, combined with proximal fine-tuning (SQYFR) and extended to a 3x-looped MoE (26.4B physical / 5.4B active), compressed thinkingmachines/Inkling-2-Mini. The text weights were reduced from 18.89 GB to 5.38 GB (1.63 bits/weight) and 8.87 GB (2.69 bits/weight). This compression utilized 4 KB SRAM hashed block-vector trellis codes (ht2/ht3), row trellis codes (rtr1/rtr4), and a tr2 floor on the tail lm_head bucket (head.7).

Methodology and Availability

The compression recipe involves post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10x, and bits allocated per row by Fisher information. This is followed by 4,000 steps of distillation, where each layer is re-quantized from the original weights. The code for this recipe is available under the MIT license, and the compressed files, such as lm2-qwen35-4b-1.2gb, are packed for the Apple Silicon runtime.

✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →

The daily brief

One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.

One email a day. Unsubscribe in one click, any time.

Today's brief

Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.

~4 min · 3 stories · Nov 21

▶ Play today's brief Listen on Spotify

New every morning, and the back catalogue is archived by date.

Reporting from

LookAtMyQuant, a quantization compiler, has demonstrated significant compression of large language models (LLMs) while maintaining performance. It reduced Qwen3.5-4B to 1.23 GB and Qwen3.8-27B to 7.2 GB, outperforming other quantization methods in benchmarks. The compiler also compressed Inkling-2-Mini from 18.89 GB to 5.38 GB, using specialized trellis codes and distillation.