Distributed systems should use state-oriented consistency, not system-wide consistency

The Problem with System-Wide Consistency

Early in the development of a clustered message broker, developers focused on selecting a single consistency model for the entire system. This approach led to a design flaw where every pod, upon booting, loaded all client session states into memory. This occurred because the system was designed to be ready to serve any client, leading to an unfiltered read of all stored session data, not just the data relevant to clients that would connect to that specific pod.

Incident: Out-of-Memory Errors

The design flaw manifested as out-of-memory (OOM) errors, with pods being killed by the kernel's OOM handler. Despite a memory limit of 512Mi, pods consumed excessive memory. Initial hypotheses, such as load-balancer imbalance, were disproven by metrics showing similar memory usage across pods even with vastly different connection counts. A pod serving 130 times more connections than another had nearly identical memory consumption, indicating a fundamental design issue rather than a load distribution problem.

Investigation Reveals Root Cause

Investigation into the code responsible for session state revealed that a persistence hook on pod boot loaded the entire fleet's client session state, not just a fraction. This meant every row returned from the initial read became a live in-memory object. The design implicitly assumed that a node should be prepared to serve any client, leading to this inefficient memory allocation.

The Solution: State-Oriented Consistency

The core lesson from this incident is that distributed systems should not seek a single consistency model for the entire cluster. Instead, the focus should be on determining the specific consistency guarantee each piece of state actually needs. Adopting this state-oriented consistency approach can simplify the overall design of distributed systems and prevent resource-related issues by tailoring data handling to its specific requirements.