● Covered by 1 source · 1 reportMedium impact1 neutral
Netflix Details In-House LLM Serving Platform Using Triton and vLLM
Key points
- Netflix uses Triton for model loading, batching, and GPU scheduling.
- vLLM handles inference and offers extensibility for custom models.
- The platform supports both CPU and GPU inference for LLMs.
- Constrained decoding is used to force model responses into specific formats.
Platform Architecture Overview
Netflix has outlined the architecture of its in-house platform for serving LLMs. This platform integrates with Netflix's existing JVM-based serving layer, which manages routing, feature retrieval, and logging. Smaller models run in-process on CPUs, while larger requests are directed to a service where Triton orchestrates model loading, batching, GPU scheduling, and multi-framework serving.
Integration of Triton and vLLM
For GPU-based inference, Netflix selected vLLM due to its operational suitability and extensibility, while Triton retains control over the serving environment. Triton manages the overall serving infrastructure, and vLLM performs the actual inference. Compatibility between Triton and vLLM versions is critical, requiring specific releases to be tested and pinned together to prevent deployment issues.
Custom Model Support and Packaging
Integrating custom models presented challenges, as vLLM's Hugging Face compatibility was insufficient for some Netflix models. The company utilized vLLM's extension points to support custom architectures and decoding behaviors. Netflix also compared Triton's Python backend and vLLM backend for packaging, finding that the vLLM-backend approach allows for more independent evolution of models and frontends.
Addressing API Gaps and Constrained Decoding
Despite Triton exposing an OpenAI-compatible API alongside KServe HTTP and gRPC frontends, Netflix encountered discrepancies in feature handling across these integrations. One specific example is constrained decoding, which enables Netflix to enforce specific output formats, such as valid JSON, by filtering the tokens a model can generate at each step.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Reporting from
Netflix has shared details about its internal platform for serving large language models (LLMs), which integrates Triton and vLLM to manage real-time and batch inference workloads across CPUs and GPUs. This platform allows Netflix to support diverse model sizes and hardware while maintaining a consistent production workflow for LLM inference.