Lumabri Enables P2P Swarm Execution of Mixture-of-Experts Models with Colibri Engine

Distributed Model Inference

Lumabri introduces a method for executing large mixture-of-experts (MoE) models across a peer-to-peer swarm. It utilizes the Colibri engine, which is written in pure C and has no external dependencies. This approach allows multiple machines to contribute to or access an AI model without requiring each machine to download the entire model upfront.

On-Demand Data Streaming

When an inference request is made, only the specific bytes of the model required for that operation are fetched from a peer. These bytes are then stored in a local mirror, ensuring that subsequent requests for the same data are served from local disk at full speed. This mechanism allows for efficient use of network resources and faster response times after the initial data transfer.

Broad Machine Compatibility

The system is designed to operate effectively on various hardware configurations, including machines without GPUs. The Colibri engine was developed with CPU and SSD performance as a primary consideration, meaning a GPU enhances speed but does not alter the fundamental operation or output. This broad compatibility aims to recruit a wider range of participants for model sharing compared to GPU-exclusive networks.

Technical Implementation

Lumabri's `serve` command runs a tracker (for indexing file holders) and a maintainer (for answering byte-range reads). The `chat` command mounts the model via `liblumabri.so`, an `LD_PRELOAD` shim that intercepts libc calls like `open` and `pread`. Missing data blocks are fetched from peers, written to a local mirror, and then processed by the engine. A local content-addressed store saves verified data blocks.