Why would Uber chase another Spark speedup after already squeezing about three and a half times lower CPU cost with Gluten and Velox?
Uber Engineering says its latest lab test of Onehouse Quanton, a commercial Spark engine that adds native vectorized execution and table-aware planning, cut steady-state write rounds by 5.5 times on a 60-billion-record fact table and 4.7 times on a 10-billion-record dimension table.
The promise matters because Uber's lake ingestion alone burns more than 200,000 cores and the platform handles over 100 PB of data read and written per day across 500,000 applications, according to Uber Engineering.
Gluten is a middle layer that offloads Spark SQL operators from the JVM to a native engine like Meta's Velox, which can use vectorized instructions that process many values at once.
That offload delivered roughly 3.5 times faster queries and 3.4 times lower CPU cost across migrated workloads at Uber, with up to 79 percent runtime improvement and up to 98 percent cost reduction on one workload that fell from 1 hour 7 minutes to 16 minutes, according to Uber Engineering.
But those gains covered SQL execution, not the heavier ingestion path where Spark reads existing data, applies inserts and updates, writes new Parquet files and maintains table metadata.
Quanton targets that write path directly, with native readers that understand Hudi merge-on-read logs and Iceberg deletes, pipelined I/O that overlaps storage access with compute, and optimizations to file creation, indexing and table maintenance, according to Uber Engineering.
Hudi is an open source table format that lets Uber treat plain Parquet datasets as mutable tables with versioned commits and indexes, and Uber manages about 17200 such tables.
In the test, Uber ran synthetic data modeled on production patterns on an isolated HDFS cluster on Kubernetes with Spark 3.5.9 and Hudi 1.1.1 on both sides, holding executor allocation identical between Quanton and open source Spark.
The headline speedups exclude periodic metadata-table compactions that occur every six to seven commits, where Quanton was only 2.8 times faster at 60.42 minutes versus 21.27 minutes for the baseline.
Even with that filter, the spread was stark, as Quanton's worst steady-state round on the fact table at 9 to 10 minutes still beat the baseline's best at 35 to 40 minutes, according to Uber Engineering.
The second question that follows is whether the win holds when you shrink the cluster, and Uber tested that by cutting the fact-table workload from 1128 vCPUs to 141 vCPUs.
Quanton's relative advantage grew as resources fell, from 5.4 times faster at 1128 vCPUs to 8.5 times faster at 141 vCPUs, and an eightfold cut in executors increased write time 7.5 times for open source Spark but only 4.8 times for Quanton, according to Uber Engineering.
In practical terms Quanton finished a round in 69.8 minutes on 141 vCPUs while the baseline needed 78.9 minutes on 1128 vCPUs, which translates to 89 percent fewer provisioned executor vCPU-hours per round under fixed allocation, according to Uber Engineering.
Phase breakdowns show where the time went, with the data-table write dropping 7.8 times from 475.15 minutes to 60.77 minutes and metadata-table writes falling 2.8 times, while deduplication and indexing ran about 9 percent slower and compaction stayed flat, according to Uber Engineering.
That uneven profile is the catch for any platform team budgeting a migration, because faster file writes alone do not erase index lookups, small-file partitioning and background maintenance.
The naming can confuse newcomers, since Nvidia's unrelated RTX Spark hardware platform, a compact GB10 system, was recently reviewed by Thurrott.com as the little engine that couldn't after testing found broad failures, which has nothing to do with Apache Spark, the distributed data engine Uber is tuning.
Uber says it contributed more than 30 patches upstream for Gluten and Velox to reach production, covering fallback for row-to-column conversions, C++ Parquet encryption and missing operators, so similar hardening would be expected before Quanton moves from lab to fleet-wide rollout.
For teams running Hudi or Iceberg ingestion at scale, the result to watch is not the peak speedup but whether those isolated write gains survive real table maintenance, mixed workloads and Drogon, Uber's Spark-as-a-Service platform that controls how every job is submitted, at full fleet scale.
