Databricks AI Serving Adapts to Any Model

Databricks has launched a new AI serving platform designed to eliminate the complexities of deploying and managing custom machine learning models in production. The system aims to automatically adapt to the unique resource needs and traffic fluctuations of any model, from small scikit-learn classifiers to large, fine-tuned LLMs.

This new AI Serving Platform tackles a core industry challenge: the wide disparity in resource profiles and traffic patterns for custom models. Unlike platforms optimized for a single foundation model, Databricks' offering must serve everything from a 2MB classifier on a single CPU to a 70B parameter LLM across multiple GPUs, each with different latency budgets and batching needs.

Traditionally, managing this variability meant significant engineering overhead for customers, involving constant re-profiling and tuning of configurations like replica counts and autoscaling thresholds. Databricks refers to this burden as the 'ML Stack Tax,' arguing it slows down innovation as valuable engineering time is spent on operational firefighting rather than developing new capabilities.

Mission: Erase the ML Stack Tax

The company's mission with its Databricks Custom Model Serving is to remove this tax across a model's lifecycle. This includes simplifying pre-production deployment by mirroring development environments, ensuring reliable, scalable, and cost-efficient production serving, and streamlining post-production observability with integrated telemetry.

This post focuses on the production serving stage, detailing how the platform achieves over 300,000 queries per second (QPS) with latency under 10 milliseconds (p99) for a broad range of models, all without manual configuration.

Architecture: Latency, Scale, and Cost Efficiency

The platform's architecture is built around three core, often conflicting, constraints: low latency, high scale, and cost efficiency. To achieve this balance for diverse models, it employs three key components.

First, a short, isolated request path minimizes latency overhead at each hop. Every serving endpoint is a dedicated Kubernetes deployment, ensuring that one endpoint's performance issues do not impact others.

Second, automatic runtime selection deploys models on the inference engine best suited for their type, whether it's a classic ML model or a large language model requiring GPU optimization.

The heart of the system is the AutoPilot Pod Autoscaler (APA), a custom Kubernetes controller. This autoscaler continuously monitors signals from load balancers and individual pods, including concurrency, queue depth, CPU/GPU utilization, and memory usage. It then makes intelligent scaling decisions in real-time.

The Autoscaler: Adapting to Model and Traffic

The APA addresses two primary sources of unpredictability: the model itself and the traffic it receives. Model resource profiles are often unknown in advance; a CPU-intensive model might serve one request per core, while an agent could handle hundreds. The APA learns each model's runtime limits and adjusts how many requests each replica should handle, a process called model-aware vertical scaling.

Traffic patterns are equally unpredictable, with sudden spikes and drops. The APA reacts instantly to shifts in demand, employing request-based horizontal scaling to add or remove replicas as needed.

This dual approach, combining model-aware vertical scaling with request-based horizontal scaling, ensures both efficiency and responsiveness. Traditional autoscalers often struggle with either efficiency (resource-based) or responsiveness (request-based), leading to over-provisioning or performance degradation during traffic surges.

The platform safeguards against metric noise by ensuring concurrency adjustments only occur when stable thresholds are met, capping changes per decision cycle, and enforcing minimum/maximum concurrency limits. Concurrency changes happen at a lower cadence (every 30 seconds) than horizontal scaling, relying on historical metrics.

Scale-up is aggressive to prevent latency issues during spikes. Incoming requests are scraped every second, and the APA makes upscaling decisions every five seconds based on traffic over the preceding 20 seconds. This approach significantly reduces queueing and HTTP 429 errors during demand surges, with customers reporting up to a 5x improvement.

This intelligent autoscaling, coupled with efficient runtimes and a streamlined request path, allows the MLflow-packaged models to operate at high throughput and low latency without constant manual intervention.