Technology

Spark Iceberg writes get a safe audit branch

Snowflake lets Spark stage Iceberg writes on branches and publish in one commit.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Spark write audit publish on Iceberg branches
Credit: Snowflake

Snowflake brought Iceberg Spark APIs to Snowpark Connect for Apache Spark with write-audit-publish on branches.

The detail that matters is one config, spark.wap.branch, which routes a DataFrameWriterV2 append to audit_run instead of main.

Publishing is a single call to cat.system.fast_forward that moves main to the branch snapshot without rewriting data files.

Readers of main see nothing until that commit, so bad rows stay isolated where Spark and other engines can query branch_audit_run like a table.

If checks fail, you drop the branch and main was never touched, which replaces shadow tables and swap-and-rename scripts with one table and one schema.

Snowflake says the path works across Snowflake-managed and externally managed tables in Polaris, AWS Glue and Unity Catalog through catalog-linked databases.

Spark jobs run on Adaptive Compute with no clusters to size, under Snowflake Horizon Catalog governance and lineage.

DataFrameWriterV2 support is in public preview and flexible partitioning with expression and bucket transforms is in private preview, per Snowflake.

Incremental reads process only changes between snapshots, while Time Travel and tags in public preview give stable checkpoints for rollback.

Type mapping still decides if that write lands cleanly, since Spark types convert to Iceberg types on both creation and writes per Iomete reference.

Klarent, a Telenav company, moved off Amazon EMR and runs 3.6x faster at 60% lower total cost with its longest pipeline down from over 10 hours to under three, Snowflake said.

Kumar Maddali, VP of Engineering and AI at Klarent, said his team assumed Snowflake would cost more and found the opposite.

The limit is preview status, with branch publishing in private preview and partitioning still rolling out.

For AI-generated pipeline changes, you get a real-data trial with an auditable snapshot before production ever sees it.

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsSnowflakeApache IcebergSparkSnowpark ConnectKlarentPolarisAWS GlueUnity Catalog

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.