NVIDIA MPS and Triton on Amazon EC2 Reduce ASR Inference Costs by 75%

Addressing ASR Inference Efficiency

Automatic speech recognition (ASR) inference on Amazon EC2 often faces challenges with low GPU utilization, where a single request uses only 15–20 percent of a GPU's capacity. NVIDIA CUDA's default time-slicing behavior further exacerbates this by forcing sequential access, leaving significant hardware idle. This inefficiency becomes critical for applications with strict latency requirements, such as Heidi Health, which processes over 2.4 million clinical consultations weekly.

Solution Implementation and Results

AWS, NVIDIA, and Heidi Health collaborated to implement a solution using NVIDIA CUDA Multi-Process Service (MPS) combined with NVIDIA Triton Inference Server on Amazon EC2 GPU instances. This integration reduced the GPU infrastructure required for ASR inference by 75%, specifically decreasing the number of instances from 16 to 4 for Heidi Health. The new setup maintained sub-second latency at a rate of 92.1 requests per second (RPS) per GPU.

The GPU Utilization Problem

A single ASR inference request using the Parakeet TDT 0.6B V2 model utilizes only about 15–20 percent of an NVIDIA L40S GPU's streaming multiprocessors. The remaining 80 percent remains idle during each forward pass. CUDA's default time-slicing mechanism allows processes exclusive GPU access sequentially, leading to context switching overhead and preventing concurrent execution. This limitation meant a single GPU could handle approximately 62 RPS at acceptable latency, necessitating 16 GPUs for Heidi Health's peak traffic demands.