rqlite creator reflects on a decade of distributed SQLite development

The Genesis of Distributed SQLite

In September 2014, rqlite was introduced as an open-source project combining SQLite with a Raft consensus log. This innovation allowed for the replication of writes across a cluster, ensuring data availability even if a node failed. At its inception, rqlite was the first open-source system to implement this specific combination.

Motivation and Early Development

The initial need for rqlite arose from a clustered system requiring a small, replicated relational storage solution. Although the immediate work project took a different direction, the creator continued developing rqlite due to the general applicability of the problem and the compact nature of the solution. The first version utilized the goraft library, later transitioning to Hashicorp's Raft implementation in 2016, which rqlite continues to use.

A key design decision from the outset was to establish rqlite as a CP (Consistency and Partition tolerance) system. This meant writes had to pass through the leader, require majority acknowledgment, and a partitioned minority would refuse writes. Subsequent features like the HTTP API, read-consistency levels, and snapshotting were built upon this foundational design.

Parallel and Subsequent Developments

While rqlite pioneered the open-source space, other projects also explored distributed SQLite concepts. Expensify had an internal Paxos-style SQLite replication system, later released as Bedrock in 2016. ActorDB, an Erlang system, also appeared around the same time as rqlite, replicating individual SQLite databases per actor using Raft. These projects demonstrated that the idea of distributed SQLite was emerging, but rqlite distinguished itself by offering a standalone, general-purpose, open-source solution.

Canonical later developed dqlite for LXD and MicroK8s, seeking an embedded replicated store without relying on etcd. Although discussions occurred regarding rqlite's suitability, dqlite was ultimately developed independently, initially in Go and then rewritten in C with its own Raft library.

The Evolving Landscape

Today, 'distributed SQLite' has become a recognized category, encompassing numerous projects, some venture-funded and others acquired. This retrospective provides historical context, tracing the origins of this category back to 2014 with rqlite's initial release, predating many contemporary discussions that often begin around 2021.