# Netflix's Real-Time Service Map Pipeline _Netflix’s real-time service map pipeline uses a streaming-first, three-stage architecture to visualize complex service dependencies at scale._ **Published:** 2026-07-13 **Source:** https://www.startuphub.ai/ai-news/technology/2026/netflix-s-real-time-service-map-pipeline --- Netflix engineers grappled with the challenge of creating a unified, real-time view of their sprawling service dependencies. Traditional batch processing methods proved inadequate, delivering data too stale for critical incident response. This led to the development of a streaming-first architecture, detailed on the [Netflix Tech Blog](https://netflixtechblog.com/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-f4b792f3f0d8?source=rss----2615bd06b42e---4). Stale Service MapsDriver traditional batch processing delivered outdated service dependency views for incident responsedrivesStreaming-First ArchitectureCorecontinuously ingests millions of network flow records and IPC metrics per secondFrom the article 2 mentionsThis led to the development of a streaming-first architecture, detailed on the Netflix Tech Blog.usesThree-Stage PipelineCoredistributed architecture processes data through initial, network resolution, and final aggregation stagesFrom the article 7 mentionsThe network layer ingestion relies on a three-stage distributed aggregation pipeline.Stage 1: Initial AggregationContextaggregates raw network flow records and IPC metrics from various sourcesReal-Time ObservabilityEffectprovides topology updates within tens of minutes for live events and rapid incident responseFrom the article 3 mentionsNetflix engineers grappled with the challenge of creating a unified, real-time view of their sprawling service dependencies.Stage 2: Network ResolutionContextresolves network intermediaries to identify direct service-to-service communication pathsFrom the articleThe IPC pipeline, unlike the network flow log pipeline, requires only a single stage because its data is already correctly partitioned at the application level, eliminating the need for intermediary resolution.Unified Service MapOutcomevisualizes complex service dependencies at scale for comprehensive system understandingFrom the article 2 mentionsThe core innovation is a distributed service map pipeline designed for scale and real-time updates.thenStage 3: Final AggregationContextenriches aggregated data with service metadata for a complete dependency view The core innovation is a distributed service map pipeline designed for scale and real-time updates. This system ingests millions of network flow records and IPC metrics per second, processing them through reactive pipelines with built-in backpressure handling. ## Streaming-First for Real-Time Observability Unlike batch systems that offer hourly or daily snapshots, Netflix’s pipeline continuously ingests data, providing topology updates within tens of minutes. This real-time capability is essential for live events and rapid incident response. Backpressure is crucial for managing the data flow. When downstream systems slow down, upstream components are signaled to pause, preventing data loss or system crashes. This allows the pipeline to degrade gracefully under load. ## A Multi-Layered Approach The architecture employs physically separate topology layers for network flows, IPC metrics, and distributed tracing. Each layer is optimized for its specific data type and query patterns, offering independent evolution and unified querying. ## The Three-Stage Aggregation Pipeline The network layer ingestion relies on a three-stage distributed aggregation pipeline. This design tackles the complexity of network intermediaries, which obscure direct application-to-application dependencies in raw flow logs. ### Stage 1: Initial Aggregation Flow logs from multi-region Kafka streams are filtered, batched into 5-minute windows, and initial aggregators are created. This stage identifies flows involving intermediaries but doesn't resolve them. ### Stage 2: Network Intermediary Resolution This stage groups flows by intermediary (like load balancers or proxies). By redistributing aggregators based on intermediary identifiers, it joins incoming and outgoing flows to resolve direct application edges, effectively removing network hops from the topology view. ### Stage 3: Final Aggregation and Enrichment Resolved edges are further aggregated, enriched with external data like ownership and health metadata, converted into graph entities, and then persisted to the graph database with throttled writes. This three-stage process, including a second redistribution step, prevents the creation of 'hot nodes' by spreading the load of resolution and enrichment across multiple instances. ## Engineering Challenges and Lessons The journey involved significant hurdles, including Kafka consumer lag, uneven traffic distribution ('hot nodes'), and memory management issues like garbage collection pauses and persistent heap pressure. Serialization complexity and stream processing inefficiencies also demanded solutions. The team learned that scale changes everything in distributed systems. Optimizing one bottleneck at a time and recognizing that distribution is key to achieving scale were paramount. For inter-stage communication, Server-Sent Events (SSE) were chosen over gRPC or message queues due to their lightweight nature, minimal serialization overhead, and natural integration with reactive streams, proving that industry best practices aren't universally applicable. The IPC pipeline, unlike the network flow log pipeline, requires only a single stage because its data is already correctly partitioned at the application level, eliminating the need for intermediary resolution. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.