# Snowflake Cuts Costs With Custom Incrementalization _Thrive Learning cut data costs by up to 99% and improved refresh times with Snowflake's custom incrementalization for Dynamic Tables._ **Published:** 2026-07-28 **Source:** https://www.startuphub.ai/ai-news/technology/2026/snowflake-cuts-costs-with-custom-incrementalization --- Thrive Learning, a UK-based enterprise learning platform, has significantly boosted the price-performance of its data pipelines by adopting [Snowflake's Dynamic Tables Custom Incrementalization](https://www.snowflake.com/content/snowflake-site/global/en/blog/thrive-dynamic-table-costs-custom-incremental). This capability allows for fine-grained control over data refreshes, directly impacting customer satisfaction and the quality of Thrive's analytics and AI products. Thrive LearningCore From the article 7 mentionsThrive Learning, a UK-based enterprise learning platform, has significantly boosted the price-performance of its data pipelines by adopting Snowflake's Dynamic Tables Custom Incrementalization.Dynamic TablesContextinitially used for automatic incremental updates, but enrichment caused duplicatesFrom the article 5 mentionsThe core challenge for Thrive involved a data pipeline, dubbed RecordStore, that initially benefited from Dynamic Tables' automatic incremental updates.Custom IncrementalizationCoreSnowflake feature allowing fine-grained control over data refresh logicFrom the article 4 mentionsThe solution lay in Snowflake's custom incrementalization feature for Dynamic Tables, allowing engineers to define explicit MERGE INTO logic over changes.Reduced Refresh TimesEffectimproved data refresh speed, directly impacting customer satisfaction and product quality99% Cost ReductionOutcomeThrive Learning cut data costs by up to 99% for their RecordStore pipelineFrom the article 6 mentionsTransformation compute dropped from approximately 150 credits/day to 5 credits/day (a 97% reduction), and auto-clustering costs fell from 150 credits/day to 2 credits/day (a 99% reduction).Improved AnalyticsEffectbetter data quality and faster access for Thrive's AI and analytics productsFrom the articleThis capability allows for fine-grained control over data refreshes, directly impacting customer satisfaction and the quality of Thrive's analytics and AI products.Two-Lane WorkaroundDriverFrom the articleTo mitigate this, Thrive implemented a two-lane approach: a 'fast lane' for recent data refreshed every six hours and a 'slow lane' for older data refreshed weekly.still causedHigh Data CostsDriverfull table refreshes for complex data pipelines drove up Snowflake expenses significantlyFrom the article 2 mentionsBeyond cost savings, custom incrementalization preserved Thrive's data governance. The core challenge for Thrive involved a data pipeline, dubbed RecordStore, that initially benefited from Dynamic Tables' automatic incremental updates. However, enriching this data with additional keys from multiple sources introduced duplicates. This complexity forced [Snowflake](/ai-news/technology/2026/snowflake-streams-for-real-time-ai) to revert to full table refreshes, driving up costs. To mitigate this, Thrive implemented a two-lane approach: a 'fast lane' for recent data refreshed every six hours and a 'slow lane' for older data refreshed weekly. While this reduced transformation costs, the subsequent UNION operation between the two lanes sharply increased auto-clustering expenses, creating a costly trade-off. ## Costly Workarounds At its peak, this pipeline consumed hundreds of credits daily, split between auto-clustering and transformation compute. Each attempted fix addressed a symptom rather than the root cause, which was the unreliable upstream Content ID. ## Custom Incrementalization Solves the Problem The solution lay in Snowflake's custom [incrementalization](/ai-news/technology/2026/snowflake-turbocharges-data-pipelines) feature for Dynamic Tables, allowing engineers to define explicit MERGE INTO logic over changes. This approach retained the efficiency of processing only changed rows, similar to Snowflake streams and tasks, but crucially kept scheduling, retries, and lag management within Snowflake's purview. This enabled Thrive to transition from full refreshes to processing only the delta, drastically reducing both transformation and auto-clustering costs. Transformation compute dropped from approximately 150 credits/day to 5 credits/day (a 97% reduction), and auto-clustering costs fell from 150 credits/day to 2 credits/day (a 99% reduction). Refresh latency for the pipeline plummeted from hours to mere seconds, as each run now only inspects the sources that have actually changed. The pipeline now scales with the rate of data change, not its total size. ## Maintaining Data Lineage Beyond cost savings, custom incrementalization preserved Thrive's data governance. By integrating the logic directly into their dbt workflows, the team avoided the need for separate orchestration layers and maintained clear lineage tracking. This ensures that custom incremental Dynamic Tables remain first-class citizens within their dbt DAG, managed with the same testing and documentation standards as other models. The success of this implementation is now leading Thrive to apply the same pattern to other pipelines facing similar complexities, promising further cost reductions and fresher data for their AI initiatives. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.