Cloudflare's celld uses deterministic simulation testing to find and reproduce bugs

Introducing celld and its Challenges

Cloudflare is building celld, a runtime designed for Cloudflare Workers and Durable Objects applications, intended to run on local machines. celld operates as a distributed system, which inherently presents challenges in achieving reliability due to potential reliance on unstated assumptions, as highlighted by Peter Deutsch's "The Eight Fallacies of Distributed Computing."

Bugs in distributed systems often depend on specific sequences of events like message delays, failed writes, or node restarts. These sequences can vary between test runs, making bug reproduction difficult. To address this, Cloudflare employs deterministic simulation testing (DST).

How Deterministic Simulation Testing Works

Deterministic simulation testing (DST) runs celld's production code within a simulator-controlled environment. The simulator manages the order of events, such as incoming requests, completed storage operations, and timer firings, ensuring that the same event-handling code used in production is tested under controlled conditions.

The simulator also controls the execution of asynchronous tasks and the behavior of object storage, including delaying or failing write operations. It can advance simulated time independently of real time, which is crucial for testing time-dependent behaviors like retries and alarms in conjunction with other events.

Reproducibility and Bug Discovery

Within configured limits, the simulator uses random choices to generate requests, select the next event to run, decide on storage operation outcomes, and introduce faults. A key aspect of DST is the use of a seed for its random-number generator. This seed ensures that with identical code, settings, and seed, the same sequence of events, intermediate states, and results are produced.

Although still under development and not publicly available, the DST system has already identified previously unknown bugs within celld. This capability allows developers to consistently reproduce failures, investigate their root causes, and verify the effectiveness of proposed fixes.