Amazon SageMaker HyperPod integrates new Ray capabilities for distributed ML workloads

Enhanced Ray Integration on SageMaker HyperPod

Amazon has announced new Ray capabilities on SageMaker HyperPod, integrating the open-source Ray framework with HyperPod's infrastructure. Ray is used by data scientists to scale distributed Python workloads across GPU clusters for tasks like distributed training with Ray Train and model serving with Ray Serve.

Simplified Management and Observability

Previously, running Ray on Kubernetes required manual configuration of YAML manifests, Docker image rebuilds, kubectl port-forwarding for the Ray Dashboard, and manual setup of Prometheus and Grafana. With this update, data scientists can now create Ray clusters, access the Ray Dashboard and Amazon Managed Grafana observability dashboards, and connect JupyterLab or Code Editor workspaces directly from SageMaker Studio. This also includes submitting distributed jobs and configuring hung job detection.

Improved Fault Tolerance and Performance

The integration provides automatic fault tolerance for Ray training jobs through HyperPod's node health monitoring and recovery features. Additionally, tiered checkpointing is available for faster resume times, utilizing HyperPod's distributed tiered storage. For model serving, SageMaker JumpStart integration allows loading model weights directly into Ray Serve endpoints, with KV cache offloading to tiered storage to support long-context requests.

Compatibility and Prerequisites

These new capabilities are compatible with open-source KubeRay and standard Ray APIs, ensuring existing scripts and workflows can run without modification. To utilize these features, users need an Amazon SageMaker HyperPod cluster with Amazon EKS orchestration, along with the SageMaker Spaces EKS add-on, HyperPod Observability EKS add-on, and the KubeRay operator installed on their cluster.