Netflix Redesigns Service Topology Pipeline for Scalability

Service Topology Redesign

Netflix has detailed the redesign of its Service Topology streaming pipeline, which creates a real-time map of service dependencies. This update focuses on the production engineering required to maintain the network-flow pipeline at scale, building on earlier descriptions of Service Topology's multi-source graph capabilities.

Multi-Source Data Integration

Service Topology integrates data from eBPF network flows, inter-process communication (IPC) metrics, and distributed traces. This allows engineers to query these layers independently or merge them for a comprehensive view of service dependencies. The system is used for incident investigation, blast-radius analysis, dependency understanding, and production change management.

Three-Stage Processing Pipeline

The updated network-flow ingestion path now processes data in three stages. The first stage consumes multi-region Kafka streams, filters invalid records, batches data, and creates initial aggregators. The second stage resolves intermediaries into direct application-to-application edges and redistributes results. The final stage enriches nodes with information like health and ownership before persisting them to the graph database.

Addressing Bottlenecks and Backpressure

The previous design concentrated work for popular destinations, leading to instances experiencing up to 100 times typical traffic during I/O-heavy enrichment. Separating resolution from enrichment and persistence redistributed this work. The pipeline now uses Apache Pekko Streams to manage backpressure, ensuring that when graph storage cannot keep up, pressure propagates upstream to pause the Kafka consumer, preventing record loss.