Snowflake Cuts Costs With Custom Incrementalization

Thrive Learning, a UK-based enterprise learning platform, has significantly boosted the price-performance of its data pipelines by adopting Snowflake's Dynamic Tables Custom Incrementalization. This capability allows for fine-grained control over data refreshes, directly impacting customer satisfaction and the quality of Thrive's analytics and AI products.

The core challenge for Thrive involved a data pipeline, dubbed RecordStore, that initially benefited from Dynamic Tables' automatic incremental updates. However, enriching this data with additional keys from multiple sources introduced duplicates. This complexity forced Snowflake to revert to full table refreshes, driving up costs.

To mitigate this, Thrive implemented a two-lane approach: a 'fast lane' for recent data refreshed every six hours and a 'slow lane' for older data refreshed weekly. While this reduced transformation costs, the subsequent UNION operation between the two lanes sharply increased auto-clustering expenses, creating a costly trade-off.

Costly Workarounds

At its peak, this pipeline consumed hundreds of credits daily, split between auto-clustering and transformation compute. Each attempted fix addressed a symptom rather than the root cause, which was the unreliable upstream Content ID.

Custom Incrementalization Solves the Problem

The solution lay in Snowflake's custom incrementalization feature for Dynamic Tables, allowing engineers to define explicit MERGE INTO logic over changes. This approach retained the efficiency of processing only changed rows, similar to Snowflake streams and tasks, but crucially kept scheduling, retries, and lag management within Snowflake's purview.

This enabled Thrive to transition from full refreshes to processing only the delta, drastically reducing both transformation and auto-clustering costs. Transformation compute dropped from approximately 150 credits/day to 5 credits/day (a 97% reduction), and auto-clustering costs fell from 150 credits/day to 2 credits/day (a 99% reduction).

Refresh latency for the pipeline plummeted from hours to mere seconds, as each run now only inspects the sources that have actually changed. The pipeline now scales with the rate of data change, not its total size.

Maintaining Data Lineage

Beyond cost savings, custom incrementalization preserved Thrive's data governance. By integrating the logic directly into their dbt workflows, the team avoided the need for separate orchestration layers and maintained clear lineage tracking.

This ensures that custom incremental Dynamic Tables remain first-class citizens within their dbt DAG, managed with the same testing and documentation standards as other models.

The success of this implementation is now leading Thrive to apply the same pattern to other pipelines facing similar complexities, promising further cost reductions and fresher data for their AI initiatives.