Data Products Are Dead; Services Are In

The era of building data products for specific, pre-defined use cases is facing a reckoning. As companies scale through acquisitions and embrace AI agents that compose data in unpredictable ways, the rigid data product model is proving to be a bottleneck. The solution, according to insights from Howden's Group Chief Data Officer Barry Panayi, lies in embracing data services. This shift is detailed in a recent Databricks blog post.

Panayi explains that a data layer structured as open, governed services offers far greater adaptability than a catalog of discrete products. This is essential when data consumers are no longer just human analysts but AI agents that will combine data in novel ways. A services layer inherently supports this emergent behavior, unlike a product catalog that requires every use case to be anticipated.

The Product Model's Limitations

Companies experiencing rapid growth, particularly through acquisition, find the one-product-per-use-case approach cumbersome. Howden, for instance, acquired more than one business per week last year. Integrating data post-acquisition previously took around six months.

This pace forced users to create data silos or pull data from disparate sources to meet immediate needs. This fragmentation led to limited adoption and duplicated efforts, highlighting the cost of slow integration cycles.

Architecturally, the shift involves moving data mastering and quality checks upstream, as close to ingestion as possible. This makes data usable much faster, drastically altering the integration timeline.

Unified Context and Reconciliation

With numerous data sources, the same metric can exist in multiple correct versions, leading to significant manual reconciliation efforts. Panayi's team previously spent considerable time identifying the appropriate data version for specific answers due to a lack of a common data model or taxonomy.

The solution involved building a standardized data model, the Accord data model, which codifies business logic. This embeds reconciliation into the data itself, rather than relying on human intervention during each reporting cycle. If your taxonomy isn't codified, your team becomes the reconciliation engine, a costly tax that scales inversely with business growth.

Scaling From Pilots to Capabilities

Many organizations struggle to scale AI pilots into production capabilities. Howden moved beyond building unique use cases from scratch by standardizing pipelines, sharing code, and creating reusable data assets on the Databricks platform. This enables cross-domain analytics by combining various data types efficiently.

The transition from isolated experiments to scalable, reusable capabilities is now tangible, supported by a unified data view. While productionizing models as consistent services is an ongoing effort, the foundation for scalable AI consumption is in place.

Insight Lag Over Data Freshness

In industries like insurance, where transactional speed isn't the primary driver, the focus shifts from data freshness to reducing insight lag. This is the time gap between when data exists and when it can be used to inform decisions.

For brokers, having the latest insights before client meetings is crucial. Previously slow, batch-driven reporting meant data was often stale. Now, brokers can access real-time insights on market trends and company performance, transforming data into a competitive advantage.

The Power of Conversational Analytics

The demand for instant answers, similar to interacting with tools like ChatGPT, drove the adoption of conversational analytics. Databricks Genie allows users to ask questions of their company's data and receive fast answers.

This capability drastically reduces the time analysts spend building dashboards for ad-hoc requests. In Howden's US retail business, Genie has saved hundreds of hours by providing direct answers, freeing up analysts for higher-value work.

The ideal scenario involves a 'Howden Intelligence Layer' that intelligently routes user questions to the appropriate service, whether it's a general AI model or a governed data query via Genie. Users shouldn't need to know the underlying data source.

Design for the Future Pace

The core advice for leaders scaling data and AI efforts is to design properly from the start, anticipating future needs. This includes collaborating with platform partners on architecture and involving process and agentic work leaders early in the design phase.

The critical takeaway is to start thinking about data services, not just data products, to build an adaptable architecture ready for the evolving demands of AI consumption. Designing for the pace you are heading toward, not just the pace you are at today, is paramount.