Running Kimi K3 Mixture-of-Experts Model on Apple M1 Max Laptops

Deltafin Project Overview

Deltafin is a research project designed to run a Mixture-of-Experts (MoE) model, specifically Kimi K3, on consumer-grade hardware. It allows the Kimi K3 model, which is larger than the machine's capacity, to operate on an Apple M1 Max laptop equipped with 64GB of RAM.

Performance and Reproducibility

While not fast, processing at approximately 16 seconds per token on an M1 Max, the setup offers exact and reproducible results. The performance can improve with newer chips and more RAM. The project emphasizes local execution for consistency.

Setup and Model Download

The setup involves three main steps: environment preparation (Python 3.12+ and Xcode CLT), building a fused MXFP4 kernel, and downloading the model. Users can choose between a full download of the model (1.7 TB) or a streaming mode, which fetches expert data as needed. The streaming option is suitable for initial trials without committing significant disk space.

Expert Data Management

The Kimi K3 model requires 25.8 GB of expert data per token (16 experts x 92 layers). Accessing this data from local disk takes about 4 seconds, while network access can take minutes. Deltafin provides tools to manage expert data, including a resumable script to fetch all experts and an option for partial downloads. The system reminds users if it's in streaming mode and indicates the cost of a full download.

Model Inference and Interaction

The project includes scripts for running inference, allowing users to ask questions in a chat format or perform raw completions. Tokens are printed as they are generated, providing real-time output. There is also an option to convert non-expert weights to INT8 format, which halves per-token I/O with no significant quality change.