Intel Reduces LLM Weight Storage to 1.485 Bits Using New BITCOS Format

New BITCOS Format for LLM Compression

Intel researchers introduced BITCOS (BITmap and COmpacted Signs), a new storage format designed to compress ternary large language model (LLM) weights. This format allows for a reduction in storage from 1.58 bits to 1.485 bits per weight without modifying the model's actual weights or requiring retraining. The approach focuses on how weights are stored rather than changing the model's structure or output.

Performance Improvements

The implementation of BITCOS resulted in notable performance gains. Decoding throughput improved by as much as 18% on CPUs and 27% on GPUs. These improvements stem from the more efficient storage and retrieval of model weights, which directly impacts the speed at which LLMs can process information.

Addressing Storage Inefficiencies

The standard 1.58-bit figure for ternary models assumes an equal distribution of the three possible weight values (-1, 0, +1). However, Intel's research found that real ternary models contain a significantly higher proportion of zero weights, ranging from 29.7% to 51.5% across various models. Traditional storage methods, such as fitting five ternary values into an eight-bit byte, do not efficiently account for these zeros, leading to an actual rate of 1.625 bits per weight due to unused space in blocks.

How BITCOS Works

BITCOS addresses this by dividing weights into two streams. One stream uses a single bit to indicate whether a weight is zero or nonzero. The second stream assigns a sign bit only to nonzero weights. This means a positive or negative weight uses two bits, while a zero weight only requires one bit (for its presence). This method allows zeros to occupy less space, effectively packing the same model contents into a smaller footprint without altering the model's output.