Encore Rebuilds Linux MicroVM Stack for Apple Silicon to Run Firecracker Locally

The Challenge of Firecracker on macOS

Encore's backend applications have utilized Firecracker microVMs for builds since mid-2022. Firecracker provides virtual machine isolation with container-like startup times by stripping down emulated hardware to essential Linux kernel needs. However, Firecracker relies on KVM, a Linux-specific virtualization module, making it incompatible with macOS, where most Encore engineers develop.

Developing a Cross-Platform Solution

To allow engineers to run the build system locally on their Macs while maintaining Firecracker in production, Encore developed "crackling." This single microVM API orchestrates Firecracker on Linux and Apple's native hypervisor on macOS. Implementing this required a significant effort to rebuild the Linux image toolchain to function correctly on macOS.

Previous Development Workflow

Before "crackling," Encore engineers accessed a shared build machine via SSH. This setup involved a script to onboard new users, configure their environment, and manage VM images and binaries. Updates to the environment required a second script to transfer cross-compiled binaries and manage image conversions, as Firecracker boots block devices while Docker produces layers.

Addressing Image Conversion

A key challenge was converting Docker layers into a bootable block device for Firecracker. Encore developed a custom conversion tool, as no existing solution was found. This tool operates partially on the developer's laptop and partially over SSH, facilitating the necessary image format transformation.