Deltafin fork runs Kimi K3 (2.8T) at 1 token/s on MacBook Pro using four SSDs

Kimi K3 on Consumer Hardware

A modified version of Deltafin, specifically the gavamedia/deltafin fork, has successfully run the Kimi K3 large language model on an Apple Silicon MacBook Pro. This implementation utilizes data streamed from four solid-state drives (SSDs) to manage the model's substantial parameter count.

The Kimi K3 model, with 2.8 trillion parameters, typically targets infrastructure with 16 nodes and 4.8 TB of aggregate VRAM. This experiment aims to demonstrate the feasibility of running such a large model on consumer-grade hardware.

Performance Metrics

The setup achieved a performance of approximately 1 token per second, specifically measured at 0.2901 tokens/s (3.447 seconds per token). This represents a 1.9% throughput increase since the last update. Earlier measurements showed lower throughputs, indicating ongoing optimization efforts.

Scaling tests with fewer drives showed reduced performance: one drive yielded 52% of the four-drive speed, two mirrors 73%, and three drives 90%.

Maintaining Model Quality

A core principle of this experiment is to maintain the full quality of the Kimi K3 model. The Deltafin fork ensures that all 16 experts of the model are utilized for every token, without any pruning or shortcuts. The model itself is the sole authority for deciding each token, preventing any reduction in output quality for the sake of speed.

Experimental Focus

This project is presented as an experiment to determine the maximum efficiency achievable when running large models like K3 on consumer hardware. It is not a product pitch but rather an exploration of the boundaries of current technology and what can be learned from such a challenging endeavor.