TrnRT is a newly released open-source inference engine designed for running large language models (LLMs) on AWS Trainium hardware. It provides an OpenAI-compatible API, allowing developers to integrate it with existing tools and workflows that support the OpenAI API standard.
The engine operates on a trn2.48xlarge instance. It implements decode operations using NKI megakernels, with approximately one kernel per layer half of the model. A key feature is its prefix cache, which spans both device and host memory to optimize performance for sequential requests.
Currently, TrnRT supports the GLM-5.3-Flash model, with plans to add more open models in the future. To use TrnRT, users need Docker, an AWS trn2.48xlarge instance for serving, and any x86_64 host for compiling the necessary components. The build process involves the Neuron SDK 2.32 and Python 3.13.
The deployment process involves building a Docker image, compiling a cache (which can take several hours on an x86_64 host without a NeuronCore), and then serving the model on the trn2.48xlarge instance. Once served, the engine can be called using any OpenAI client by pointing to the local API endpoint.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
TrnRT, an open-source inference engine, has been released for large language models on AWS Trainium, providing an OpenAI-compatible API. It uses NKI megakernels for decode operations and manages a prefix cache across device and host memory, initially supporting GLM-5.3-Flash.