PyTorch 2.15 Improves MPS Performance on macOS, Adds FP8 Support

MPS Backend Enhancements

The PyTorch team at Hugging Face has focused on improving the MPS backend to make macOS a more capable platform for running and training PyTorch models. This work involves identifying performance bottlenecks and unsupported operations in MPS, then implementing fixes upstream in PyTorch, often by replacing MPSGraph with dedicated Metal kernels.

Performance Gains for LLMs

PyTorch 2.15 delivers substantial speedups for large language models (LLMs) on macOS. Mixture-of-Experts (MoE) models now decode 5-6.5 times faster than in version 2.14, with Qwen3.6-35B-A3B reaching 61.3 tokens/s on an M3 Ultra. This improvement is due to native grouped matrix multiplication, fewer GPU-CPU synchronizations, and faster top-k routing. Additionally, single-row linear layers use specialized matrix-vector kernels, and softmax over the vocabulary is approximately 8 times faster.

New Features and Memory Optimization

The update introduces FP8 support, allowing float8_e4m3fn values to be stored in half the space of BF16 and enabling scaled matrix multiplication. For video generation, the memory footprint of decoding a video with Wan2.2's tiled VAE has been reduced from 270 GiB to a peak of about 8 GiB. Other improvements include more linear algebra operations on the GPU, such as complex LU and triangular solves, faster small-matrix inverses, and quicker QR decomposition.

System Requirements and Future Outlook

PyTorch 2.15 now requires macOS 15 or later and Python 3.11 or later. Looking ahead to version 2.16, the focus will be on fully integrating attention on Metal, further migrating MPSGraph to Metal, and achieving even faster LLM decoding.