AWS introduces Ray Serve Deep Learning Containers for TorchServe inference workloads

TorchServe Maintenance Discontinuation

TorchServe, a tool for model inference, is no longer actively maintained. The project has no planned updates, bug fixes, new features, or security patches, and vulnerabilities may not be addressed. This discontinuation means users are responsible for managing the entire dependency chain, including compatible PyTorch and CUDA versions, security patching, and debugging.

Introduction of Ray Serve Deep Learning Containers

AWS has introduced Ray Serve Deep Learning Containers (DLCs) to address the challenges posed by TorchServe's lack of maintenance. These DLCs are pre-built, performance-optimized Docker images designed for serving models behind an HTTP endpoint. They bundle a deep learning framework, its dependencies, and the GPU stack into a tested and patched combination.

Components and Benefits of Ray Serve DLCs

The Ray Serve DLCs are built on Amazon Linux 2023 (CPU variant) or NVIDIA Amazon Linux 2023 (GPU variant, including CUDA runtime libraries). They include PyTorch, the Ray Serve serving layer with FastAPI and Uvicorn, and utilities for vision, audio, and multimodal workloads, such as FFmpeg with NVIDIA hardware acceleration. AWS maintains and tests these containers, simplifying dependency management and security for users.

Deployment Example

The announcement includes a guide for deploying a vision-language model on Amazon Elastic Kubernetes Service (Amazon EKS) using the new Ray Serve DLC. This process involves running the model with the DLC, running the serving application with Ray Serve, and deploying it on a single GPU node. The complete code for this deployment is available in an accompanying repository.