← All stories
● Covered by 1 source · 1 reportLow impact1 neutral

TrnRT Open-Source LLM Inference Engine Released for AWS Trainium

🔄 Updated 4d ago
New to BrevFeed? We gather this story from every outlet covering it into one summary — ranked by real-world impact, not just the latest headline — so you never miss what matters. What is BrevFeed? →

Key points

  • TrnRT is an open-source LLM inference engine for AWS Trainium.
  • It offers an OpenAI-compatible API from a trn2.48xlarge instance.
  • Uses NKI megakernels for decode and a cross-memory prefix cache.
  • GLM-5.3-Flash is the first supported model.

Introduction to TrnRT

TrnRT is a newly released open-source inference engine designed for running large language models (LLMs) on AWS Trainium hardware. It provides an OpenAI-compatible API, allowing developers to integrate it with existing tools and workflows that support the OpenAI API standard.

Technical Architecture

The engine operates on a trn2.48xlarge instance. It implements decode operations using NKI megakernels, with approximately one kernel per layer half of the model. A key feature is its prefix cache, which spans both device and host memory to optimize performance for sequential requests.

Model Support and Requirements

Currently, TrnRT supports the GLM-5.3-Flash model, with plans to add more open models in the future. To use TrnRT, users need Docker, an AWS trn2.48xlarge instance for serving, and any x86_64 host for compiling the necessary components. The build process involves the Neuron SDK 2.32 and Python 3.13.

Deployment and Usage

The deployment process involves building a Docker image, compiling a cache (which can take several hours on an x86_64 host without a NeuronCore), and then serving the model on the trn2.48xlarge instance. Once served, the engine can be called using any OpenAI client by pointing to the local API endpoint.

✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →

The daily brief

One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.

One email a day. Unsubscribe in one click, any time.

Today's brief

Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.

~4 min · 3 stories · Nov 21

▶ Play today's brief Listen on Spotify

New every morning, and the back catalogue is archived by date.

Reporting from

TrnRT, an open-source inference engine, has been released for large language models on AWS Trainium, providing an OpenAI-compatible API. It uses NKI megakernels for decode operations and manages a prefix cache across device and host memory, initially supporting GLM-5.3-Flash.