● Covered by 1 source · 1 reportMedium impact1 positive
Cloudflare prototypes Zstandard compression in Pingora to save petabytes of cache storage
Key points
- Cloudflare prototyped Cache Transcoding using Zstandard (zstd) in Pingora.
- The system compresses eligible assets to one-third of their original size.
- It reduces storage needs and inter-data center bandwidth.
- The trade-off is a small increase in CPU usage for encoding.
Cache Transcoding with Zstandard
Cloudflare has prototyped a system called Cache Transcoding, which integrates Zstandard (zstd) compression into its Pingora proxy. This system encodes eligible assets with zstd before writing them to disk, keeping them compressed while in cache and during transfer between data centers via Tiered Cache. Assets are then decoded before being served to clients.
Storage and Bandwidth Savings
Initial testing showed that this encoding shrunk eligible assets to one-third of their original on-disk size. This provides Cloudflare with petabytes of effective cache capacity and reduces data transferred between data centers. The encoding cost is paid once when an asset enters the cache, with storage and bandwidth savings realized every time the asset is reused.
CPU Trade-off
The implementation involves a minor increase in CPU cost in the origin-facing proxy for the compression process. Zstandard, developed by Yann Collet at Facebook and open-sourced in 2016, is a lossless compression algorithm designed to balance compression ratio with speed. The prototype uses zstd level 3 to achieve significant compression benefits without creating a CPU bottleneck during cache fills.
Impact on Cloudflare Operations
With increasing memory and storage costs, making efficient use of deployed memory is critical for Cloudflare's distributed storage products, including its CDN. This Cache Transcoding system allows Cloudflare to expand its effective cache capacity and continue serving customers efficiently by optimizing storage and bandwidth usage.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Reporting from
Cloudflare prototyped a Cache Transcoding system using Zstandard compression within its Pingora proxy, reducing cached asset sizes by two-thirds on average. This system aims to expand effective cache capacity and decrease cross-data center bandwidth usage by trading a minor CPU increase for significant storage savings.