Llama-modchip Enables 284B DeepSeek-V4-Flash on 64GB RAM Without GPU

Enabling Large Models on System RAM

Llama-modchip is a collection of ggml/llama.cpp kernel patches, quantization transcoders, and runtime tools. Its primary function is to enable the execution of sparse Mixture-of-Experts (MoE) checkpoints directly within system RAM on x86_64 and ARM64 processors, eliminating the need for a dedicated GPU.

This tool specifically allows the 284B DeepSeek-V4-Flash model to fit into 64 GB of physical RAM. Standard builds of DeepSeek-V4-Flash typically exceed this memory limit, leading to performance issues or out-of-memory errors. Llama-modchip reduces the checkpoint size from 76.87 GiB to 56.56 GiB, ensuring the model and a 256K KV cache can reside entirely in RAM.

Performance Optimizations

The project includes CPU cache and SIMD kernel patches that significantly improve prefill and decode speeds. These patches replace single-column quantized matrix-vector loops with a 4-level register and cache tiling schedule, resulting in a 77% increase in prefill speed and a 53% increase in decode speed. For example, on a 16-core Zen 5 system, prefill speed increased from 25.0 to 44.36 tokens per second.

Further optimizations include register-stationary MQA8 FlashAttention and non-temporal MoE streaming. The latter prevents cold expert weights from displacing hot KV cache lines from the L3 cache, maintaining efficient data access.

Advanced Features for Model Interaction

Llama-modchip incorporates self-speculative drafting and instant agent context restore capabilities. It pairs a 1.64 MB L2/L3 Zobrist n-gram drafter with /dev/shm KV slot snapshots, allowing for rapid context restoration (13.1–32.6 ms for 40-tool Claude Code prompts). An O(1) speculative rollback fix is also included for deepseek4 recurrent/MLA states.

Quantization and Memory Management

The tool utilizes specific quantization techniques to achieve its memory footprint reduction. DeepSeek-V4-Flash, with its 284.33B total parameters and 13.17B active parameters across 43 MLA layers and 256 routed experts, is compressed to 1.71 bits per weight (bpw). This compression is crucial for fitting the model within the 64GB RAM constraint.

By default, DeepSeek-V4-Flash evaluates all 6 routed experts plus 1 shared expert per layer (Top-6) without expert dropping, maintaining full model accuracy. Users can optionally set LLAMA_MOE_TOP_K to 5 or 4 to trade a small perplexity delta for higher decode throughput.