Real-time applications incur hidden costs from duplicated state across distributed systems

Hidden Costs of Distributed State

Real-time applications frequently integrate various tools, such as Kafka for events, Flink for processing, Redis for serving state, and a database for records. While this modular approach offers flexibility and independent scaling, it introduces costs related to data movement and the maintenance of overlapping state across different components. These costs are often not reflected in direct memory bills but impact system complexity and performance.

Unnecessary Data Copies

While some data copies are essential for fault tolerance (replicas), recovery (checkpoints), and history preservation (event logs), others are maintained primarily because processing occurs in one system and applications consume results from another. For example, a velocity counter calculated by Flink might be published to Redis for a fraud-scoring service, as Flink's internal state is not designed for general-purpose serving. This creates a separate serving layer with its own operational overhead.

Synchronization Challenges

Maintaining synchronized state becomes more complex in applications spanning multiple availability zones. If a payment system serves reads from asynchronous replicas, an update (like a refund) might not propagate instantly to all replicas. A service reading from an outdated replica could then make incorrect decisions, such as issuing a duplicate refund, highlighting the need for careful application design when relying on asynchronously replicated state.