Databricks Tackles GPU Woes

Databricks is detailing its strategy for maintaining GPU reliability across its AI platform, a critical challenge as organizations scale demanding workloads like foundation model training. The company points to three primary failure categories: outright job crashes, subtle performance degradations that go unnoticed, and numerical corruption leading to incorrect results. These issues can cripple expensive, time-consuming AI training runs.

GPU Failure Modes at Scale

Crashed jobs are the most apparent, often manifesting as NCCL watchdog timeouts. However, diagnosing the root cause requires tracing across hardware, fabric, and software layers. More insidious are silent slowdowns caused by degraded GPUs or network links, which waste compute resources without immediate alerts. Numerical corruption, stemming from memory faults or software errors, can lead to incorrect models discovered only post-training.

This is a critical challenge for anyone running large-scale distributed GPU training.

Stress Testing with Real-World Workloads

To proactively identify these failure modes, Databricks AI leverages its own demanding workloads. These include reinforcement learning for agents, large-scale coding models, and document intelligence systems. Such diverse and intensive tasks expose fabric flakiness, thermal hotspots, and collective communication edge cases before they impact broader customer deployments.

One incident highlighted a NCCL timeout caused by a single Infiniband port flap. The issue underscored the critical interplay between the InfiniBand transport layer's timeout (NCCL_IB_TIMEOUT) and higher-level timeouts, demonstrating how even brief network interruptions can derail long training jobs.

A Multi-Stage Health Check System

Databricks has developed a system called 'gpu-monitor' to address these issues across the entire node lifecycle. Bootstrap checks validate hardware integrity before workloads begin, covering GPU compute, connectivity, memory health, and fabric bandwidth.

Nodes failing these initial checks are immediately quarantined. Passive continuous checks then monitor for non-deterministic failures that emerge under load, such as NVLink lane status degradation or GPU clock throttling. Nodes exhibiting issues are cordoned and re-tested.

Periodic multi-node active checks further validate inter-node fabric behavior. These tests assess NCCL collective bandwidth across various payload sizes, identifying subtle performance bottlenecks that single-node checks might miss.

This rigorous, multi-layered approach aims to ensure the stability and accuracy of GPU-accelerated AI development on the Databricks platform.