TrnRT: An Open-Source LLM Inference Engine for AWS Trainium, and What Its Token Costs Mean
Published
TrnRT is an Apache-2.0 inference engine for AWS Trainium2, released by RunAnywhere on November 17, 2026. It serves GLM-5.3-Flash through an OpenAI-compatible API from one trn2.48xlarge, reporting 3.57 million tokens a minute and $0.040 per million at spot prices. It supports one model, and its benchmarks are self-run.
What TrnRT is
TrnRT, short for Trainium RunTime, is an Apache-2.0 inference engine for large language models on AWS Trainium2. RunAnywhere, a Y Combinator Winter 2026 company, posted it as a Show HN on November 17, 2026, saying the team had started three weeks earlier. It serves an OpenAI-compatible API, with chat, completions, streaming, tool calls and reasoning output, from a single trn2.48xlarge instance: 16 Trainium2 chips with 96GiB of HBM each.
It supports exactly one model today: Z.ai's GLM-5.3-Flash, a 320 billion parameter mixture-of-experts model with 18 billion active parameters, served in FP8 with a 1M token context. The box runs two engines, each tensor-parallel over 32 logical NeuronCores, behind a session-affine router so a conversation keeps hitting the same prefix cache. Every compute graph is compiled offline on an ordinary x86 host, a step the README says takes hours, and the serving box loads the results.
Why the team built it
In the HN post, the founder explained the gap. A large MoE model with a million-token context needs a lot of accelerator memory, and AWS's own stack did not run this one: vLLM Neuron's latest release registered no GLM model, and NxDI, AWS's distributed inference library, is no longer under active development. A commenter added that SGLang's recent Neuron update targets Trainium3, while the founder described Trainium2 as the only generation open for public use.
The technical core is NKI megakernels. NKI is AWS's Neuron Kernel Interface. Rather than chaining many small kernels with XLA glue between them, TrnRT runs decode as roughly one fused kernel per layer half. RunAnywhere's write-up argues decode on this hardware is limited by small instructions, kernel boundaries and exposed collectives rather than memory bandwidth, so each kernel keeps its working set in on-chip SBUF, uses both physical cores evenly and moves cross-chip communication inside the kernel. It reports a batch-8 decode step falling from 32.2ms to 24.2ms between its October 17 and November 1 builds.
What it means for token cost
The cost of a self-hosted token is simple division: the instance's hourly price over the tokens it processes in an hour. RunAnywhere ran SemiAnalysis's AgentX benchmark, which replays real coding-agent sessions, with 32 concurrent agents at a 1M token context. At its default decode batch of 12, the README reports 3.57 million tokens a minute, about 59,500 a second, and 28.9 requests a minute. At a spot price of $8.60 an hour in us-east-2 on November 3, 2026, that works out to $0.040 per million tokens. On a capacity block at $35.76 an hour, it is $0.167.
Read those figures carefully. The token count includes cached input, and replayed agent sessions are dominated by long, repeated prompts that hit the prefix cache. That is the right workload for agents, but it is not comparable to a price list that charges output tokens separately. Per-user decode at batch 12 was 22.3 tokens a second, and median time to first token on a cached prompt was 6.53 seconds under that load. Throughput and responsiveness trade against each other: batch 4 gave 46.6 tokens a second per user at a higher cost per token.
The strongest structural argument is memory price. The write-up quotes AWS capacity block rates of $23.84 per terabyte-hour of accelerator memory for trn2.48xlarge against $48.69 for an eight-GPU H200 p5e.48xlarge. For big MoE models with long contexts, memory capacity, not compute, decides how many sessions fit on a box.
The caveats
The benchmarks are self-run, one hour per point, and the write-up says plainly they are not official InferenceX results. They were measured on an earlier internal build; the public repository reproduces the same graphs and returned byte-identical outputs on 384 of 384 test requests. Sparse-attention prefill uses an approximation affecting about 1.2 percent of block selections, and no task-level accuracy evaluation was run on the benchmarked build. Batch sizes above 12 were not measured, and at 48 agents the host-memory prompt cache became the limit.
There is also a tension worth noticing. On HN a founder wrote that the team could show better tokens per dollar on Trainium2 than on H200. The blog post itself says it makes no claim about Trainium2 against GPUs per request, and its only GPU figure comes from a different workload. Until a like-for-like comparison exists, treat the cost-per-memory gap as established and the per-request advantage as unproven.
Practically, TrnRT is one model, one instance type and one SDK version, Neuron 2.32, with hours of compile time for every change of shape. Commenters on HN asked for tokens per second rather than per minute and for an SGLang comparison; one called the thin Trainium software ecosystem the real story.
Who should care
Teams serving GLM-5.3-Flash at volume, especially agent workloads with long shared prefixes, have a concrete, reproducible path to cheap tokens without waiting for GPU capacity. Everyone else should watch for the second supported model, because an engine that serves one model is a proof of concept for the approach, not yet a platform.
Cheaper tokens also do not change the cheapest token, which is the one never sent. Prefix caching is exactly that idea inside an engine. RDK applies it one level up: agents retrieve a few hundred tokens of indexed, encrypted context from your vault through MCP instead of reprocessing whole documents, which cuts token spend 80 to 90 percent on repeated or reference-heavy work, on any engine.
Frequently asked questions
- What is TrnRT?
- TrnRT is an open-source, Apache-2.0 LLM inference engine for AWS Trainium2 from RunAnywhere, released on November 17, 2026. It serves an OpenAI-compatible API from one trn2.48xlarge using fused NKI megakernels for decode and a prefix cache across device and host memory. GLM-5.3-Flash is the only supported model so far.
- How cheap is inference on TrnRT?
- RunAnywhere reports $0.040 per million tokens at a spot price of $8.60 an hour, and $0.167 on a capacity block at $35.76 an hour, on the AgentX coding-agent benchmark at batch 12. Those token counts include cached input, the benchmarks are self-run, and the figures are not directly comparable with per-token API price lists.
- Is Trainium cheaper than NVIDIA H200 for inference?
- Per terabyte of accelerator memory, yes on AWS's capacity block prices: $23.84 per terabyte-hour for trn2.48xlarge against $48.69 for an eight-GPU H200 instance. Per request, it is unproven. RunAnywhere's own write-up makes no like-for-like GPU comparison, although a founder claimed better tokens per dollar on Hacker News.
- Can TrnRT run models other than GLM-5.3-Flash?
- Not yet. The README lists GLM-5.3-Flash as the first and only supported model, with more open models described as coming soon and a model request template in the repository. The architecture documentation explains what a new model family needs, but each one requires its own kernels and a compile cache that takes hours to build.