Homelabber Runs 2.8-Trillion-Parameter AI Model on CPU with Discontinued Optane Memory

Running Large AI Models on Homelab Hardware

A homelab enthusiast on Reddit has managed to run Moonshot AI’s Kimi K3, a 2.8-trillion-parameter AI model, using only a 40-core 3rd-gen Xeon Scalable CPU. This was achieved by utilizing discontinued second-generation Intel Optane PMem 200 memory sticks, bypassing the need for a GPU for inference. The project is a follow-up to an earlier build that used first-generation Optane memory.

Optane Memory for Capacity and Cost Savings

The Kimi K3 model is designed for data center inference engines and requires at least 610GB of RAM for even the smallest community quantizations. With rising DRAM prices, using secondhand Optane memory provides a cost-effective way to achieve the necessary memory capacity at home. Intel's Optane Persistent Memory 200 Series (PMem 200), launched in 2020, offers capacities up to 512GB per stick and can deliver higher memory bandwidth than its predecessor.

Performance and Configuration Details

The setup achieved an inference speed of 0.85 tokens per second with 1-bit quantization, which dropped to 0.75 tokens per second with 3-bit quantization. This is considerably slower than typical inference speeds, with a 50,000-token task estimated to take 13 hours. The build used Optane in App Direct Mode, which exposes it as a separate, persistent region, giving more control over tensor placement. The previous build, which used a smaller model and an RTX 3060 GPU, achieved around 4 tokens per second.

Thermal Management Challenges

A significant challenge encountered was the heat generated by the Optane DIMMs. The builder emphasized the necessity of active DIMM cooling, employing dual 12W blower fans directed at the modules to maintain acceptable operating temperatures.