Stripe Automates Database Remediation Using Graph Search and State Machines

Automated Database Incident Recovery

Stripe's engineering team developed a system to automate database incident recovery. This system models the company's global infrastructure as a graph, using graph search algorithms and state machines to automatically compute and execute remediation plans. This initiative addresses challenges with previous hard-coded remediation systems that struggled with scalability and required manual intervention.

Impact on Operations

The new system dynamically adapts to different MongoDB shard layouts, leading to a significant reduction in operational overhead. Database-related pager alerts have decreased by approximately 30%, which translates to 200 fewer pages per year. Additionally, the system has eliminated an estimated 12 days of unhealthy shard states annually, improving overall database stability.

Addressing Previous Limitations

Stripe's prior remediation system, based on hard-coded plugins, faced issues with fragile dependencies, complex multi-failure scenarios, and unhandled intermediate states. This often necessitated manual intervention. For instance, in a six-month period, the control plane paged operators 124 times for misconfigured shards and 32 times for single-node-down scenarios, blocking critical operations for an average of one hour per incident.

Graph-Based Remediation Logic

The new approach models MongoDB infrastructure as a graph, where nodes represent components, edges show relationships, and attributes describe states. By using graph traversal, initially Breadth-First Search (BFS) and later Dijkstra's algorithm, the system identifies valid and cost-optimized recovery paths. This allows the remediation logic to adapt automatically to evolving infrastructure and different database layouts, moving away from fixed workflows to composable rules with explicit state transitions.

Future Expansion

Stripe plans to extend this framework beyond failure recovery. Future applications include automating topology changes, facilitating blue-green deployments, and orchestrating planned maintenance alongside reactive healing. This expansion aims to further enhance the automation and resilience of Stripe's infrastructure operations.