Amazon SageMaker HyperPod has introduced new capabilities to detect and recover from silent data corruption (SDC). SDC is a hardware issue where components, such as GPUs, produce incorrect numerical results without reporting any errors. This can lead to prolonged training with corrupted data, resulting in weaker models or unexplained divergences, as observed during Meta's Llama 3 405B pre-training where SDC caused job interruptions.
Previously, SageMaker HyperPod's health monitoring agents and deep health checks could identify and replace nodes with reported faults. However, SDC bypasses these existing mechanisms because the hardware does not register any errors. The new detection methods are integrated directly into the training job itself to identify these silent failures.
Three new checks have been added to the PyTorch DistributedDataParallel (DDP) training loop to identify SDC. These include a non-finite gradient check to catch NaN, Inf, and overflow errors during forward or backward passes. A ring replay audit recomputes micro-batches from other nodes to compare loss and gradient fingerprints. Additionally, a weight fingerprint comparison checks for corruption in the optimizer step and stored weights by comparing replicas after updates.
When one of these checks triggers, the suspect rank is mapped to its corresponding node. SageMaker HyperPod then contains the corruption and replaces the identified faulty node. This process ensures that training can continue with reliable hardware, mitigating the impact of SDC.
The new SDC detection and recovery system was tested using software-injected faults on a SageMaker HyperPod cluster. The tests were conducted on ml.p5.48xlarge instances, each equipped with 32 NVIDIA H100 GPUs. The full code for this solution is available in a GitHub repository.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
Amazon SageMaker HyperPod now includes enhanced infrastructure observability features to diagnose GPU training issues more quickly. This new capability integrates diagnostic and profiling tools for training workloads on Amazon EKS and Slurm orchestrators, helping identify performance bottlenecks like unhealthy GPUs or network saturation.
Amazon SageMaker HyperPod introduced new features for workload recovery and infrastructure checks to improve the goodput of training workloads. These additions address failures that occur even when individual nodes appear healthy, such as subtle GPU faults or stalled jobs. The new capabilities include multi-node training validation, Kubernetes resource monitoring, and hung job detection.
Amazon SageMaker HyperPod now includes new detection and recovery mechanisms for silent data corruption (SDC) in distributed training. These features address SDC, which occurs when hardware returns incorrect but plausible numbers without error, by integrating checks directly into the PyTorch DistributedDataParallel training loop.