Characterization Tests Capture Undocumented Software Behavior

The Challenge of Undocumented Behavior

Software systems in production for several years often develop undocumented behaviors, quirks, and edge cases that users or other services come to depend on. These unintended dependencies are not covered by typical regression tests, which are written to validate intended functionality. This gap can lead to unexpected regressions when changes are made to the system.

Hyrum's Law and Its Implications

Hyrum's Law, named after Hyrum Wright, posits that once an API is widely used, all observable behaviors of the system, not just the documented contract, will eventually be depended upon by someone. This includes aspects like the order of results, exact error messages, or response times. Fixing what appears to be an obvious bug can inadvertently break integrations built on these 'wrong' behaviors.

Introducing Characterization Tests

Michael Feathers, in 'Working Effectively with Legacy Code', introduced the concept of characterization tests. These tests are designed to capture the current, actual behavior of a system before any modifications are made. The goal is not to assert correctness, but to document the existing state, even if it appears buggy, to create a safety net that detects any future shifts in behavior.

How Characterization Tests Work

Instead of defining what the code *should* do, characterization tests record what the code *does* do. This involves running the code and pinning its output in a test. If a behavior seems incorrect, it is still pinned, perhaps with a note. This method provides a mechanism to detect when changes to the code alter existing, even if unintended, functionality, thereby preventing regressions that stem from undocumented dependencies.