● Covered by 1 source · 1 reportMedium impact1 positive
Multiverse AI uses Ising model for LLM pruning, improving Llama-3.3-70B-Instruct compression
Key points
- New LLM pruning method uses constrained binary optimization and Ising models.
- Method maps block removal to a disordered spin system.
- Achieved 23 percentage point MMLU gain on Llama-3.3-70B-Instruct at 50% compression.
- Accounts for inter-block dependencies, unlike mean-field methods.
New Pruning Method for LLMs
Multiverse AI introduced a new method for compressing large language models (LLMs) by treating block removal as a constrained binary optimization (CBO) problem. This problem maps directly onto an Ising glass, a disordered spin system. This allows for efficient evaluation of potential pruned model configurations.
Improved Compression Performance
The new technique demonstrated significant improvements in deep compression scenarios. When applied to Llama-3.3-70B-Instruct, the method achieved a nearly 23 percentage point gain on the MMLU benchmark at 50% compression, outperforming existing block-removal methods.
Addressing Inter-Block Dependencies
Traditional block-removal methods often assess each block independently, similar to mean-field approaches in physics. However, transformer blocks are not independent; removing one block can affect the impact of removing others. The Multiverse AI method accounts for these interactions, or couplings, between blocks, which is crucial for maintaining model quality during aggressive pruning.
Leveraging Statistical Physics Tools
The challenge of searching exponentially large configuration spaces with pairwise couplings is a domain where statistical physics tools are effective. By assigning a binary variable to each transformer block and using a second-order Taylor expansion of the model's loss, the method can approximate the Hessian matrix, which helps quantify individual block importance and inter-block relationships.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub CompactifAI/Block_removal_through_constrained_binary_optimization GitHub CompactifAI/Block_removal_through_constrained_binary_optimization.Reporting from
Multiverse AI developed a new method for pruning large language models (LLMs) by reformulating block removal as a constrained binary optimization problem, mapping it to an Ising glass model. This approach allows for ranking candidate configurations without extensive benchmarking, achieving a 23 percentage point gain on MMLU for Llama-3.3-70B-Instruct at 50% compression compared to other block-removal methods. The method accounts for inter-block dependencies, which traditional mean-field approaches overlook.