Netflix's Autoscaler Evolution

Netflix operates over 30,000 Apache Flink jobs, a scale that makes efficient resource management, particularly autoscaling, non-negotiable. The streaming giant detailed its journey with two distinct autoscaling systems in a recent Netflix Tech Blog post, revealing a strategic shift towards adopting and contributing to open-source solutions.

The company initially built an in-house autoscaler around 2019. This system functioned like an external observer, monitoring Flink jobs via cluster-level metrics from Netflix's telemetry platform, Atlas. It was designed as a streaming job itself, making it scalable and independent of Flink's internal workings. This approach successfully reduced resource usage by 25-45% for simpler, single-operator pipelines.

However, this external view had limitations. It treated entire clusters with coarse container metrics and scaled a single knob, the total TaskManager count. This proved insufficient for the increasingly complex, multi-operator, stateful Directed Acyclic Graphs (DAGs) used for services like personalization and advertising. These jobs often had intricate dependencies and state management that the external observer couldn't effectively reason about. Furthermore, reliance on external metrics meant the system could miss subtle issues, as evidenced by a networking migration that led to inaccurate metric reporting, only discovered later in production.

Reasoning from Within

The turning point came with the maturation of the Apache Flink community's own autoscaler. Unlike the homegrown solution, the open-source Apache Flink Autoscaler reasons from inside the job. Its core innovation is estimating the True Processing Rate (TPR) of each operator. By analyzing how much time an operator spends actively processing versus being backpressured or idle, it can extrapolate its maximum potential throughput. This allows the autoscaler to calculate the precise parallelism needed for each vertex in the job graph, rather than scaling the entire job or cluster uniformly.

This internal perspective unlocked capabilities the first autoscaler lacked. It could handle complex, stateful DAGs and allowed individual jobs to carry their own scaling configurations, such as stabilization periods and specific thresholds, tailored to their unique workloads. This made it a natural fit for custom jobs that teams were previously scaling manually.

Making Open Source Work at Netflix Scale

Adopting the open-source autoscaler wasn't just a matter of plugging it in. Netflix had to adapt it to its internal infrastructure, which doesn't rely on the Kubernetes Operator for Flink. The core logic was refactored into a standalone library, integrated into Netflix's existing control plane orchestrated by Temporal. Each Flink job with autoscaling enabled runs its own long-running workflow, pulling metrics, evaluating scaling decisions, and actuating changes.

Key engineering challenges were overcome to make this work at Netflix's scale. Enhancements were made to Flink's JobManager to cache metric names and enable server-side filtering, allowing the autoscaler to handle jobs with up to 3,000 Flink subtasks, a significant improvement from its previous ~1,000 subtask limit. Custom logic was also added to preserve forward chaining between vertices and to respect sink limits by detecting asynchronous sink backpressure. Safety checks were implemented to prevent scaling down during critical events like region failovers or to ensure sufficient disk space for checkpoints.

The Bottom Line: Efficiency and Stability

The migration to the open-source autoscaler has yielded significant results. One team, the client telemetry and logging group, reported a 58% reduction in annualized Flink compute expenditures, equating to approximately $1.1 million in savings. This efficiency stems from dynamic adaptation to traffic cycles, continuous resource adjustment, and improved resource utilization through uniform container dimensions.

Netflix also learned that aggressive scaling down can be detrimental. By setting a lower target utilization (0.45 compared to the community default of 0.7), they trade marginal efficiency for greater stability. Fewer, calmer rescales are preferable for large, stateful jobs. The article notes that the primary cost in scaling stateful jobs today remains the restart and state recovery process, a challenge Flink 2's disaggregated state architecture aims to address.

StartupHub.ai data indicates that Apache Flink, a key component in this story, scores 50/100 overall, with verified financials showing it raised $100M in 2026. While Flink itself is a powerful stream processing engine, its effective utilization at scale hinges on sophisticated operational tooling like autoscalers. Netflix, with its 81/100 StartupHub score, demonstrates how large enterprises must innovate or adopt solutions to manage complex infrastructure efficiently. Competitors to Flink, such as JOKR (63/100) and Fountain (49/100), also compete in the broader data processing and real-time analytics space, highlighting the continuous drive for performance and cost optimization.

Why This Matters

Netflix's experience with its Flink autoscalers offers valuable lessons for any organization managing large-scale data processing. It highlights the fundamental trade-offs between build vs. buy, external vs. internal monitoring, and the critical importance of choosing the right metrics. For developers and platform engineers, the journey underscores the benefits of adopting and contributing to open-source projects, even while adapting them to unique operational contexts. The move towards a single, community-driven autoscaler simplifies Netflix's operational surface area and allows it to benefit from broader community innovation, a strategy that often proves more sustainable than maintaining bespoke internal solutions long-term.