Finding Resilience Bugs in Systems Before They Are Built

Proactive Bug Detection

Hillel Wayne describes his work in identifying bugs in systems that are still in the design phase. Companies engage him to review system concepts and predict potential failures that could impact customer experience. This method aims to catch critical issues before any code is written or deployed.

Challenging Traditional Agile Approaches

Wayne notes that this proactive bug-finding approach initially felt contrary to his background in extreme programming and agile methodologies. These methodologies often emphasize finding bugs through testing completed code, sometimes even in production environments. His work demonstrates that early design analysis can also be effective.

The 'Thing Doer' Example

To illustrate the process, Wayne uses an abstract 'thing doer' example. In its initial design, the 'thing doer' checks if a task is marked done, marks it done, then performs the task. He points out a bug where a failed task, upon retry, would see itself marked done and never actually complete the action.

A revised design, where the task is done before being marked complete, introduces a new bug: the 'two-thing doer' problem. If two instances run concurrently, one might complete the task, but before marking it done, the second instance could start and perform the same task again, leading to duplication.

Real-World Implications

Wayne states that the 'thing doer' is an abstract representation of common system designs he encounters. He has observed similar bugs in real-world applications, such as package delivery systems sending duplicate items or scheduling systems double-booking slots. This highlights the practical relevance of identifying these resilience issues early in the design process.