Technology

zstd pulls are faster, defaults are not

Crusoe says zstd cuts cold pulls up to 3x, but the win needs new digests and containerd 2.2 tuning.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

3 min read

GPU server node pulling large container image layers
Credit: Crusoe Blog

Crusoe says zstd cuts cold image pulls 2.4 to 3.0 times versus gzip. No venue prices that claim, odds are absent, volume is zero and nothing moved. That absence is the tell for a benchmark with no adoption market. Not financial advice. Markets move fast.

The obvious read is convert to zstd and pocket minutes on every cold start.

I doubt it is that clean. Crusoe said a 3.6 GB CUDA image fell from 51.5 seconds to 17.4 seconds after conversion in its managed Kubernetes tests. Crusoe said a 24.3 GB NeMo image fell from 394 seconds to 152 seconds in the same three-trial cold-pull setup. Crusoe put pod startup gains at 2.2 to 2.4 times in fresh-pod tests with caches cleared.

Crusoe is right about where time goes. Crusoe measured unpack at about 80 percent of pull time on 7 GB and 13 GB images, with containerd unpacking one layer at a time by default. Crusoe said moving the registry in-region saved only 8 percent, 51.5 seconds versus 56.2 seconds from Nvidia public. Compression is the bottleneck.

The conversion tax lands on push and provenance. Crusoe said the Go encoder in skopeo and BuildKit compresses near 180 MB/s while the native zstd CLI hits near 620 MB/s on the same data, so bulk work needs precompressed blobs. Crusoe timed conversion of a 7 GB image near 4 minutes end to end. Crusoe said each convert mints new digests, so pins, signatures and allow-listed digests must be updated.

Runtime support is broad but not free. Crusoe said any node on containerd 1.5.0 or newer unpacks zstd with no changes. Crusoe said Docker before v23 with classic storage fails with invalid tar header, and Slurm with Pyxis/enroot needs enroot 4.0.0 or newer. Crusoe said pull-through caches cannot transcode because recompression breaks digests.

Parallel unpack is the second lever, and it is version gated. Crusoe said raising downloads alone changed nothing because containerd max_concurrent_unpacks defaults to 1 and does not exist before containerd 2.2. Crusoe said 8 downloads with 8 unpacks cut a 7.4 GB zstd cold pull from 82 seconds to 28 seconds. A Docker Compose report with the containerd image store shows the risk in the other direction, with 21 parallel unpacks of a shared 5.7 GB image on Jetson Thor logging about 17,600 Write failed errors, per the issue tracker.

Stacked, the numbers hold. Crusoe said a 13 GB image fell from 183.8 seconds on gzip defaults to 41.5 seconds on tuned zstd, a 4.43 times gain, with format alone at 1.94 times and tuning alone at 2.42 times. Crusoe measured raw decompression at 681 MB/s versus 95 MB/s for gzip, about 7.2 times, with 4 times less CPU and images about 18 percent smaller. Crusoe said below about 100 MB gains fade to fixed overhead.

Convert your three largest images, re-sign them, then touch concurrency.

Mentioned in this story

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsCrusoeNVIDIA CUDANVIDIA NeMoPyTorchTensorRTcontainerdzstdBuildKit

StartupHub.ai Staff

Written by

StartupHub.ai Staff

Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Written by

Daniel Singer

Editor, reviewed this article

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.