Neutrino-1 8B Model Uses Ternary-Family Format for Efficient AI Inference

Introduction of Neutrino-1 Models

The Neutrino-1 models have been announced, featuring a proprietary ternary-family format. These models are served by a single engine from one artifact, compatible with datacenter GPUs, Apple silicon, and desktop CPUs.

Neutrino-1 8B Technical Specifications

Neutrino-1 8B is an 8.19B-parameter decoder-only transformer, distributed as a 3.88 GB file. Its 252 transformer linears are stored in a proprietary ternary-family weight format, which is eight times smaller than fp16. The weights remain bit-packed at rest and are decoded within the matrix kernels, ensuring no fp16 or fp32 weight material is stored in the decode path.

Impact on Serving Economics and Performance

The small weight size of Neutrino-1 8B changes serving economics. Single-stream decode performance is improved due to fewer bytes moved per token, allowing it to achieve higher rates than a 16 GB fp16 artifact on the same memory system. The entire model fits alongside its KV cache on an 8 GB GPU or a 16 GB laptop. The same container serves all supported platforms without conversion.

The model uses grouped-query attention, holding the KV cache at a quarter of the query width, 144 KiB per token at fp16. A 4k-token session requires 0.60 GB of cache in addition to the 3.88 GB of weights. Performance benchmarks show 24.9 tokens/second on an Apple M5 (CPU only, 9 threads) and 30.7 tokens/second on an NVIDIA L4.

Weight Coding Details

Only the transformer linears utilize the coded format. The two embedding tensors remain int8 because their rows are read one token at a time, not multiplied against the full activation stream. Normalization weights are not coded due to their small size. Approximately one-third of the file size is attributed to the vocabulary.

Across the 6.95B coded weights, 62.63% are zero, with the remainder split between 18.68% positive and 18.69% negative values, indicating a sign-balanced distribution. The balance varies by depth; gate and down feed-forward projections show 70-72% zeros in early layers, while attention projections maintain around 62% code density across all layers.

Deployment and Accessibility

The download is a coded transport of the container, not a compressed copy of an fp16 model, ensuring bit-exact expansion to the executable file. This single file runs consistently on both datacenter GPUs and laptops. Installation is facilitated via `pip install fermion-research`, which downloads the container and the platform-matching native binary, with CPU runtimes available for macOS arm64 and Linux x86-64, supported by a bit-exact torch reference path.