Uber's Data Abstraction Layer

Navigating the labyrinth of evolving data models and product needs is a perennial challenge for engineering teams. Uber faced this head-on, developing a Data Abstraction Layer (DAL) to streamline how its myriad applications access and utilize data. This Uber Engineering initiative aims to abstract away the complexities of underlying datasets, much like programming abstractions hide implementation details.

The core problem: data consumers, whether internal tools or advertiser-facing dashboards, often grapple with disparate table versions and changing schemas. This tight coupling between queries and data topology makes updates slow and arduous. The DAL introduces a unified interface, allowing data producers to evolve their models independently while consumers focus on the data they need, not how it's stored.

Simplifying Advertiser Reporting

A prime use case for the DAL is Uber's advertiser reporting. Advertisers need near real-time and historical performance data for their campaigns. Supporting the flexibility required for diverse reporting formats, charts, tables, configurable time ranges, and metrics, with traditional methods was a monumental task.

Before the DAL, building a new advertiser report could take weeks, sometimes months. The DAL, implemented as an RPC service, sits between data consumers (like front-end UIs) and data producers. It handles request routing, query orchestration, and response assembly.

Behind the Abstraction

At its heart, the DAL uses logical tables, definitions of data shape without specifying location. Metrics within these logical tables list candidate physical tables. A sophisticated table resolution process then determines the optimal physical tables and columns to query based on the request, time range, and data availability.

This involves several phases: schema eligibility, dataset availability, and column continuity. Once resolved, a query engine generates and executes database-specific queries, potentially across different systems like Uber's internal OLAP database and Apache Hive™. The engine can even retrieve data from multiple sources simultaneously.

Result assembly then stitches together the individual query responses. This can involve concatenating temporally adjacent datasets, joining complementary data, or performing rollups and scalar transformations. The end result is a coherent dataset delivered to the consumer.

Impact and Future Directions

The impact on advertiser reporting has been dramatic. Report generation time has plummeted from weeks to under two days. This agility has enabled more sophisticated, dynamic tools for advertisers, from user-specified filters to full report builders, greatly simplifying Uber ads data management.

The DAL, initially focused on ads, is expanding into other domains, demonstrating its flexibility. Future enhancements include more database integrations, support for asynchronous requests for larger data volumes, and extensibility mechanisms for domain-specific needs, addressing broader data integration challenges.