Redis client bug identified due to flaky tests during CI process

Background on Flaky Tests

Flaky tests are common in extensive codebases and can hinder development workflows. They create uncertainty, causing teams to spend time diagnosing failures that may not reflect actual issues in the code. In this instance, the redis-client library experienced significant instability, prompting urgent investigations.

Discovery of the Bug

The issue began on a Tuesday when the engineering team noticed multiple flaky tests blocking their deployment. David, an engineering manager, initiated a Slack thread and used the Test Engine dashboard to analyze reliability. A significant drop in reliability for specific tests was identified, estimated to have occurred days prior to the failures.

Link to Redis Upgrade

An upgrade to the Redis gem had been implemented a few days earlier without any immediate production issues noted. The correlation between the upgrade and the flaky tests led the team to suspect that the new version may contain the memory corruption bug, which warranted further examination.

Challenges of Debugging Non-Deterministic Bugs

Debugging non-deterministic bugs poses significant challenges since they require replicating specific conditions that may not consistently trigger failures. The primary focus of existing debugging tools is often on capturing the system state at failure points rather than addressing the root causes of corruption.