← All stories
● Covered by 1 source · 3 reportsMedium impact3 neutral

Amazon SageMaker HyperPod Adds Silent Data Corruption Detection and Recovery

🔄 Updated 1d ago — new reporting from AWS Machine Learning Blog
New to BrevFeed? We gather this story from every outlet covering it into one summary — ranked by real-world impact, not just the latest headline — so you never miss what matters. What is BrevFeed? →

Key points

  • SageMaker HyperPod now detects silent data corruption (SDC).
  • SDC involves hardware returning wrong numbers without error.
  • New checks integrate into PyTorch DDP training loops.
  • Detection includes non-finite gradient, ring replay audit, and weight fingerprint comparison.
  • Corrupted nodes are identified and replaced by HyperPod.
  • SageMaker HyperPod introduced workload recovery and infrastructure checks.
  • New capabilities include multi-node training validation, Kubernetes resource monitoring, and hung job detection.
  • The health monitoring agent (HMA) watches each node for GPU errors.
  • Deep health checks test nodes during provisioning.
  • Automatic node recovery reboots or replaces faulty hardware.
  • Enhanced observability diagnoses GPU training issues faster.
  • Observability integrates diagnostic and profiling tools for Amazon EKS and Slurm orchestrators.
  • Diagnostic summary uses CloudWatch's DetectGroupOutliers API.
  • Diagnostic summary compares GPU utilization, memory, temperature, power, and SM clock.
  • Static diagnostic rules surface GPU hardware faults like XID errors and ECC memory errors.

Addressing Silent Data Corruption

Amazon SageMaker HyperPod has introduced new capabilities to detect and recover from silent data corruption (SDC). SDC is a hardware issue where components, such as GPUs, produce incorrect numerical results without reporting any errors. This can lead to prolonged training with corrupted data, resulting in weaker models or unexplained divergences, as observed during Meta's Llama 3 405B pre-training where SDC caused job interruptions.

Limitations of Existing Health Checks

Previously, SageMaker HyperPod's health monitoring agents and deep health checks could identify and replace nodes with reported faults. However, SDC bypasses these existing mechanisms because the hardware does not register any errors. The new detection methods are integrated directly into the training job itself to identify these silent failures.

New Detection Mechanisms

Three new checks have been added to the PyTorch DistributedDataParallel (DDP) training loop to identify SDC. These include a non-finite gradient check to catch NaN, Inf, and overflow errors during forward or backward passes. A ring replay audit recomputes micro-batches from other nodes to compare loss and gradient fingerprints. Additionally, a weight fingerprint comparison checks for corruption in the optimizer step and stored weights by comparing replicas after updates.

Fault Containment and Recovery

When one of these checks triggers, the suspect rank is mapped to its corresponding node. SageMaker HyperPod then contains the corruption and replaces the identified faulty node. This process ensures that training can continue with reliable hardware, mitigating the impact of SDC.

Testing and Implementation

The new SDC detection and recovery system was tested using software-injected faults on a SageMaker HyperPod cluster. The tests were conducted on ml.p5.48xlarge instances, each equipped with 32 NVIDIA H100 GPUs. The full code for this solution is available in a GitHub repository.

Updates

🕒 2026-11-20 · new reporting from AWS Machine Learning Blog
  • Enhanced observability diagnoses GPU training issues faster.
  • Observability integrates diagnostic and profiling tools for Amazon EKS and Slurm orchestrators.
  • Diagnostic summary uses CloudWatch's DetectGroupOutliers API.
  • Diagnostic summary compares GPU utilization, memory, temperature, power, and SM clock.
  • Static diagnostic rules surface GPU hardware faults like XID errors and ECC memory errors.
🕒 2026-11-19 · new reporting from AWS Machine Learning Blog
  • SageMaker HyperPod introduced workload recovery and infrastructure checks.
  • New capabilities include multi-node training validation, Kubernetes resource monitoring, and hung job detection.
  • The health monitoring agent (HMA) watches each node for GPU errors.
  • Deep health checks test nodes during provisioning.
  • Automatic node recovery reboots or replaces faulty hardware.

✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →

The daily brief

One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.

One email a day. Unsubscribe in one click, any time.

Today's brief

Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.

~4 min · 3 stories · Nov 21

▶ Play today's brief Listen on Spotify

New every morning, and the back catalogue is archived by date.

How outlets covered it

Amazon SageMaker HyperPod now includes enhanced infrastructure observability features to diagnose GPU training issues more quickly. This new capability integrates diagnostic and profiling tools for training workloads on Amazon EKS and Slurm orchestrators, helping identify performance bottlenecks like unhealthy GPUs or network saturation.

Amazon SageMaker HyperPod introduced new features for workload recovery and infrastructure checks to improve the goodput of training workloads. These additions address failures that occur even when individual nodes appear healthy, such as subtle GPU faults or stalled jobs. The new capabilities include multi-node training validation, Kubernetes resource monitoring, and hung job detection.

Amazon SageMaker HyperPod now includes new detection and recovery mechanisms for silent data corruption (SDC) in distributed training. These features address SDC, which occurs when hardware returns incorrect but plausible numbers without error, by integrating checks directly into the PyTorch DistributedDataParallel training loop.