RealDiff Tool Identifies Runtime Behavior Changes in Pull Requests Across Six Languages

Detecting Hidden Behavior Changes

RealDiff is a new tool designed to identify runtime behavior changes in software that are not apparent through standard source code review. It operates by building two Git revisions, observing their test executions, and learning a noise baseline from multiple runs. The tool then reports the first changed behavior within each call tree.

Beyond Source Code Diffs

While a source code diff shows what lines were edited, RealDiff reveals the actual impact of those edits, including effects in files that were not directly touched by the pull request. This capability is crucial for catching subtle issues, such as a refactor in one part of the codebase inadvertently altering behavior in a seemingly unrelated component.

Illustrative Example: Sorting Stability

An example provided demonstrates how a one-line change from `List.Sort` to `OrderBy` in an infrastructure helper, which alters sorting stability, can lead to different results in an unedited pricing engine. In this scenario, a discount engine returned a different value, and two out of three tests executing this code did not assert on the change, highlighting a potential bug that RealDiff would flag.

Language Support and Prerequisites

RealDiff supports six programming languages. For .NET, it requires Git, .NET 8 SDK, and PowerShell 7. Java analysis needs a JDK and a Maven/Gradle wrapper, while Node analysis requires Node.js and the package manager specified in its lockfile. A public .NET demo pull request is available for inspection.