GKE Pod Snapshots Reduce AI Model Load Times by up to 89%

GKE Pod Snapshots Now Generally Available

Google has announced the general availability of GKE Pod snapshots, a feature designed to reduce the startup latency of AI/ML workloads. This functionality is available on Google Kubernetes Engine clusters running version 1.35.3-gke.1234000 or later. Benchmarks show significant improvements, with an 8B parameter model loading in 15 seconds and a 70B parameter model loading in 37 seconds, representing up to an 89% reduction in startup time.

How Pod Snapshots Work

Pod snapshots capture the complete running state of a workload, including CPU and GPU memory, open file descriptors, threads, CPU registers, and the container root filesystem. This allows a new replica to resume execution from the saved state without undergoing the initial model loading and initialization phases, which are typically time-consuming for large AI models. The underlying technology enabling this is gVisor, which necessitates Pods to run within the GKE Sandbox environment. Autopilot clusters include gVisor by default, while Standard clusters require a node pool with gVisor enabled.

Configuration and Management

The system uses two custom resources for configuration: PodSnapshotStorageConfig, which specifies the Cloud Storage bucket for snapshot data, and PodSnapshotPolicy, which defines how Pods are selected for snapshots (by label), sets the trigger (workload or manual), and manages retention policies via lastAccessTimeout and a cap on snapshots per group. An agent on each node handles the snapshot lifecycle, and a control plane controller manages the cleanup of obsolete snapshots.

Real-World Impact and Industry Reaction

Codeway, a Google customer, reported that GKE Pod snapshots reduced the startup time for their Retake platform from one minute to just 8 seconds, allowing them to optimize H100 instance usage. While the capture mechanism is seen as compelling, practitioners have raised questions regarding snapshot invalidation and compatibility keys, considering factors like model digest, CUDA/driver versions, and GPU topology. Google's documentation addresses some of these concerns by indicating that GKE builds a hash from the Pod's essential runtime.