New compatibility layer accelerates LLM inference in macOS VMs on Apple Silicon by up to 16x

Accelerated LLM Performance in macOS VMs

Cua has introduced a new process-scoped compatibility layer designed to enhance the performance of Large Language Model (LLM) inference within macOS virtual machines (VMs) running on Apple Silicon. This layer integrates with Apple's Virtualization.framework to expose newer Metal fast paths, which were previously inaccessible in standard macOS guest environments.

Significant Speed Improvements

Benchmarking on an M1 Ultra chip demonstrated substantial gains. TinyLlama 1.1B, when processed through llama.cpp, showed an 11.08 times increase in prompt processing speed and a 16.36 times increase in token generation speed compared to a stock VM. Prompt processing reached 98% of bare-metal performance.

Google's Gemma 4 12B QAT model also benefited, with prompt processing improving by 7.20 times and token generation by 14.54 times. The unlocked VM achieved 99.59% of bare-metal prompt speed and 94.82% of bare-metal generation speed for this model.

Addressing Virtualization Limitations

Apple's Virtualization.framework presents a macOS guest with a virtual graphics device, where the host executes Metal work on the physical GPU. This paravirtualization approach differs from direct GPU passthrough methods found in other virtualization stacks like QEMU and KVM. The new compatibility layer addresses a practical gap by enabling better utilization of the virtual GPU for Metal workloads, specifically improving LLM performance.

Research Release for Community Collaboration

The work is released as a research release under a permissive license, similar to Lume and Cua. This allows other developers and researchers to reproduce the results and identify which Apple Silicon chips, macOS versions, and Metal workloads benefit most from this compatibility layer. Source code, build scripts, a capability probe, and raw benchmark logs are provided for inspection and reproduction.