● Covered by 1 source · 1 reportMedium impact1 neutral
Amazon SageMaker HyperPod introduces tiered KV cache with Curvine for LLM inference
Key points
- Tiered KV cache extends to shared NVMe pool.
- Uses Curvine for distributed cache filesystem.
- Achieved up to 2.7x TTFT improvement in tests.
- Enables running workloads on lower-cost instances.
Addressing LLM Inference Challenges
Running large language model (LLM) inference at scale typically involves a trade-off between accommodating a growing KV cache and accepting slow time-to-first-token (TTFT). Existing solutions often require expensive GPU instances or lead to recomputation of identical prompts, increasing infrastructure costs and degrading user experience. This issue is particularly relevant for deploying foundation models like Qwen, Llama, and DeepSeek across various applications.
The Problem with Current Caching
During generation, vLLM stores attention keys and values in a KV cache to avoid recomputing them. Prefix caching reuses this cache for requests with shared leading tokens. However, on cost-efficient instances, memory for prefix caching is limited, especially with larger models or higher concurrency. This results in low cache hit rates for long prompts and isolated caches across horizontally scaled vLLM replicas, leading to cold starts when routing to different replicas.
Introducing Tiered KV Cache with Curvine
Amazon SageMaker HyperPod has implemented a tiered KV cache architecture that extends the cache hierarchy to include a shared, distributed NVMe pool. This architecture utilizes HyperPod's Managed Tiered KV Cache and Intelligent Routing capabilities, integrating Curvine as a lightweight distributed cache filesystem for the shared L2 tier (GPU to CPU to shared NVMe). This setup enables KV cache reuse across replicas at near-local-disk speeds.
Implementation and Performance
The implementation involves enabling HyperPod Tiered Storage, deploying Curvine workers on node-local NVMe, and patching the Inference Operator for filesystem-backed L2. In a test deployment, this architecture achieved up to a 100 percent cross-Pod cache hit rate and up to a 2.7x TTFT improvement. Cross-node L2 read latency was approximately 56 ms for a 1,900-token prompt. This allows workloads that previously required P5 instances to run on lower-cost alternatives.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Reporting from
Amazon SageMaker HyperPod now supports a tiered KV cache architecture using Curvine, a distributed cache filesystem, to extend KV cache beyond GPU and CPU memory into a shared NVMe pool. This development allows for KV cache reuse across replicas, improving time-to-first-token (TTFT) and reducing infrastructure costs for large language model (LLM) inference.