WASTE Engine Runs 2.78 Trillion Parameter Kimi K3 Model on Consumer Laptop

Running Kimi K3 on Consumer Hardware

The WASTE inference engine has successfully run the full 2.78 trillion parameter Kimi K3 model on a consumer MacBook Pro with 64 GB of RAM. This was achieved by keeping the model trunk in memory and streaming selected experts directly from disk, utilizing the remaining RAM as a bounded expert cache. The demonstration showed the model consuming 29 GB of RAM and processing at approximately 0.51 tokens per second.

Technical Approach

WASTE is an embeddable inference engine written in C with no third-party runtime dependencies. It was specifically developed to address the challenge of Kimi K3's size, which is 1.42 TB as published and 982 GB after conversion, making it too large for current mainstream consumer system RAM. Since a mixture of experts model activates only about 4% of itself per token, WASTE keeps the inactive weight on disk in a layout where each expert requires one read operation, streaming only what is needed for each token.

Performance and Significance

While the engine is functionally correct, with layers validated against a PyTorch reference, its current speed is slow at half a token per second. However, the significance lies in demonstrating the ability to run a model of this scale on a single consumer machine by streaming from disk. This is presented as a novel achievement, as similar trillion-scale NVMe streaming demonstrations or 671B-class recipes typically assume server-grade hardware with terabytes of DDR5 memory. The project suggests that the remaining challenges are now engineering-focused rather than fundamental feasibility.