Amazon has announced the public availability of SGLang-Neuron. This new offering is accessible through an AWS Deep Learning Container, developed by RadixArk in collaboration with Annapurna Labs. SGLang-Neuron allows for the deployment of AI models and the generation of reinforcement learning rollouts using SGLang on AWS Trainium.
AWS Trainium is a series of AWS AI chips designed for AI training and inference at scale. With SGLang-Neuron, users can now run SGLang on Trainium using their familiar SGLang workflow. This involves launching models via SGLang’s server interface and invoking endpoints with existing generation APIs. The underlying implementation of SGLang-Neuron operates on native PyTorch.
SGLang-Neuron functions as an out-of-tree plugin that connects SGLang with AWS Trainium. The plugin registers the neuron platform through SGLang’s entry points, activated by setting SGLANG_PLATFORM=neuron. While SGLang core handles API server, request lifecycle, RadixAttention, and prefix-cache bookkeeping, the plugin implements Trainium-specific models, attention, mixture of experts (MoE), and the graph runner.
TorchNeuron registers Trainium as a native PyTorch device. SGLang-Neuron utilizes standard PyTorch operations for model execution and incorporates kernels written with the Neuron Kernel Interface (NKI) for performance. These operations and kernels are compiled into executable programs using torch.compile, which the graph runner then prepares and executes during serving.
SGLang-Neuron also includes support for PD disaggregation. In this setup, a prefill server calculates the prompt KV cache and transfers it to a decode server using NIXL over Elastic Fabric Adapter (EFA). The decode server subsequently generates the remaining tokens. Both servers maintain SGLang’s scheduler and request lifecycle, with the plugin managing Trainium execution and KV-cache transfer.
To deploy models using SGLang-Neuron, users can utilize a Trainium3 Amazon EC2 instance with a pytorch-inference-sglang-neuronx container. The process involves starting the container, launching the SGLang model server within it, and then sending requests to the model server.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
Amazon announced the public availability of SGLang-Neuron, enabling users to serve AI models and generate reinforcement learning rollouts with SGLang on AWS Trainium. This integration allows developers to use their existing SGLang workflows and APIs, extending support to AWS's AI chips for training and inference.