125M-parameter transformer model autocompletes piano performances on-device

On-Device Piano Autocompletion

A developer successfully trained a 125-million-parameter transformer model to autocomplete piano performances. This model operates in real-time on an iPhone 15, processing approximately 108 notes per second. The project, named RollTab, is available as a free application for users with a MIDI keyboard and an iPhone or iPad.

Development Process and Optimizations

The development involved extensive experimentation over nearly a year. Significant improvements in the model's performance were achieved through careful selection of the MIDI representation, aggressive cleaning of training data, and the application of DPO (Direct Preference Optimization) during post-training. These steps were crucial for enabling efficient on-device operation.

MIDI Representation Challenges

Training a transformer on musical data required converting MIDI events into a discrete sequence. A naive tokenization approach, directly mapping MIDI events to tokens, would result in a large vocabulary due to the combinations of pitch and velocity. For example, 128 pitches and 128 velocity values could lead to over 16,000 tokens for note-on and note-off events alone. A more efficient approach involves factoring the representation with a grammar, such as separating NOTE_ON, PITCH, and VELOCITY into distinct tokens, to reduce the output space and improve model learning.

Impact on Creative AI

This project showcases the feasibility of deploying complex AI models for creative tasks directly on consumer mobile devices. By enabling real-time piano autocompletion, it provides a practical example of how AI can assist musicians and potentially open new avenues for interactive music creation without relying on cloud processing.