llm-d vs NVIDIA Dynamo vs AIBrix: Choosing a Self-Hosted LLM Inference Stack in 2026

llm-d, NVIDIA Dynamo, and AIBrix are the three leading open-source, Kubernetes-native stacks for serving large language models across distributed GPU clusters. llm-d reached v1.0 at the PyTorch Conference on October 20, 2026, a good moment to look past the release notes at the question teams actually have: if you are serving models on your own GPUs, which of these should you build on? This guide covers what these systems do, when you should and should not run one, and how the three compare.

What does a distributed LLM inference stack actually do?

Serving an LLM at scale means handling two very different jobs on every request. Prefill reads the whole prompt and builds the KV cache. It is compute-bound and bursty. Decode then generates tokens one at a time off that cache. It is memory-bandwidth-bound and runs longer. Put both on the same GPU and one phase starves the other.

Every serious stack in this space is built on the same handful of ideas to fix that:

• Prefill/decode disaggregation. Run the two phases on separate GPUs so each scales on the hardware it needs.
• Prefix-cache-aware routing. Send a request to the replica that already holds its shared prompt prefix in KV cache, so you skip recomputing it. Agent and RAG traffic, which resends large shared context, is where this pays off most.
• KV cache offloading. Spill cache to CPU memory or across nodes instead of recomputing it or dropping it.
• Workload- and hardware-aware autoscaling. Scale replicas on live traffic shape, not on CPU utilization.

The point worth internalizing before you compare anything: all three options below implement these same primitives. None of them invented these ideas, and each project's marketing will make its version sound singular. It mostly is not. The differences that decide your choice are organizational, not conceptual, and that is what the rest of this covers.

Should you self-host LLM inference at all?

Before any framework choice, the honest filter. Self-hosted distributed inference pays off for some teams and is pure overhead for others. Four questions settle it:

1. Workload shape. Do your prompts actually reuse context? Agents, RAG, and long system prompts reuse a lot, which is exactly what prefix-cache-aware routing exploits. Short, one-off prompts reuse little, and most of this machinery earns you less.
2. Scale economics. Self-hosting beats a managed API only past real volume, and only after you price in the engineering time to run it. Below that line, a hosted endpoint is cheaper and far less work.
3. Team. Someone has to own Kubernetes, GPU scheduling, and inference tuning together. If no one owns that today, it is a hire or a partner, not a weekend project.
4. Constraints. Data residency, sovereignty, or air-gapped requirements can force self-hosting regardless of the economics. If that is you, the question is not whether but which.

If you answered no to the first three and have no hard constraint, a managed endpoint is very likely the right call, and that is a perfectly good engineering decision rather than a lesser one. If you are not sure where you land, it is worth talking it through before you commit either way. The rest of this guide is written for the teams who answered yes.

llm-d vs NVIDIA Dynamo vs AIBrix: how do they compare?

All three are open-source and Kubernetes-native, and all orchestrate inference engines like vLLM rather than replacing them. The table below is the quick version; the notes after it are the judgment.

What is llm-d?

llm-d is the vendor-neutral option: a Kubernetes-native framework built on vLLM and the Gateway API Inference Extension, co-founded by IBM, Red Hat, Google, and CoreWeave, and a CNCF sandbox project since March 2026. Its v1.0 release, announced at the PyTorch Conference on October 20, 2026, hardens it for Day 2 operations with multi-tenant QoS, zero-downtime upgrades, a stable v1 config API, and agent-aware serving. Choose it if you want foundation-governed infrastructure, run mixed accelerators (NVIDIA, AMD, Intel, TPU), and already live in the Kubernetes Gateway API world.

What is NVIDIA Dynamo?

NVIDIA Dynamo is the datacenter-scale play: an open-source (Apache 2.0) orchestrator for serving large models across multi-node NVIDIA clusters. It pairs disaggregated prefill/decode with multi-tier KV offload (GPU to CPU to disk, via its KV Block Manager) and NIXL, NVIDIA's own transfer library, and stays engine-agnostic across TensorRT-LLM, vLLM, and SGLang. Choose it if you run substantial NVIDIA fleets and want the vendor's own scheduling and transfer stack, accepting that the gravity pulls toward NVIDIA hardware.

What is AIBrix?

AIBrix is the cost-and-efficiency control plane: an open-source (Apache 2.0) Kubernetes stack for vLLM built by ByteDance and now part of the vLLM project, proven at consumer scale. It adds LLM-specific autoscaling, high-density LoRA management, a distributed KV cache, prefix-aware routing, and prefill/decode disaggregation behind an OpenAI-compatible gateway. Choose it if vLLM is already your engine and your priority is squeezing cost out of heterogeneous GPU fleets, especially with many LoRA fine-tunes.

Which stack should you choose, and when?

A short decision guide, not a benchmark:

• Choose llm-d if you want vendor-neutral, foundation-governed infrastructure, you are standardizing on the Kubernetes Gateway API Inference Extension, and you run mixed accelerators (NVIDIA, AMD, Intel, TPU).
• Choose NVIDIA Dynamo if you serve large models on substantial NVIDIA fleets and want datacenter-grade scheduling and KV transfer from the vendor that makes the hardware.
• Choose AIBrix if vLLM is already your engine, cost efficiency on heterogeneous GPUs is the goal, and features like high-density LoRA serving match how you deploy.
• Still deciding the category? Re-read the gate above. The framework is the easy part. Whether to self-host at all is the expensive decision.

One caveat, stated plainly: these projects move fast and borrow from each other. Treat any single-source benchmark, including a vendor's own, as a hypothesis to test on your workload, not a verdict.

What makes self-hosted inference hard to run?

Picking the framework is the first ten percent. The parts that actually consume teams rarely show up in a release post:

• Operational surface area. Each of these is a distributed system running on Kubernetes running on GPU scheduling. That is three layers of failure modes, and whoever debugs it has to understand all three.
• KV cache behavior under real traffic. Prefix-cache-aware routing is only as good as your actual context-reuse pattern. Teams routinely overestimate theirs and undersee the cache evictions that quietly erase the gains.
• Autoscaling that respects GPUs. Scaling inference is not scaling web servers. Cold starts load tens of gigabytes of weights, and naive autoscaling thrashes expensive hardware.
• Upgrade cadence. Fast-moving projects ship frequent, sometimes breaking, releases. Someone owns staying current without destabilizing production.

None of this argues against self-hosting. It is the case for going in clear-eyed about what you are taking on. If you are earlier in that journey, our walkthrough on deploying LLMs on Kubernetes is a solid starting point.

What are the top use cases for self-hosted LLM inference?

Distributed self-hosting is not a default; it earns its keep in specific workloads. These are the ones where it consistently pays off, and the ones we most often get called in to build:

• Agentic workflows. Multi-step agents resend large, shared system prompts and tool context on every hop. Prefix-cache-aware routing reuses that KV cache instead of recomputing it, which is where disaggregated serving shows its biggest wins.
• Retrieval-augmented generation (RAG) at scale. RAG prepends sizable retrieved context to each query. When many requests share overlapping context, prefix caching and KV reuse cut both latency and GPU cost materially.
• High-density LoRA and multi-tenant model serving. Teams serving dozens or hundreds of fine-tunes off a shared base model get the most from LoRA-aware scheduling and a shared KV cache, packing many adapters onto the same fleet.
• High-volume, sustained inference. Past a real and continuous token volume, self-hosting on your own GPUs undercuts per-token API pricing — but only after you price in the engineering time to run it.
• Data-sovereign, regulated, or air-gapped deployments. When data residency, sovereignty, or air-gap requirements rule out a managed API, self-hosting is the only path, and the question becomes which stack rather than whether.
• Latency- and cost-sensitive production SLAs. Workload-aware autoscaling and prefill/decode disaggregation let you hold tail latency while keeping expensive accelerators busy, instead of over-provisioning for peak.

How can CloudRaft help?

CloudRaft does forward-deployed engineering: we embed with your team to build and run self-hosted inference, and we stay framework-neutral, because the right stack depends on your traffic, your hardware, and your team, not on a preference of ours. The help maps straight onto the use cases above:

• Running agents or RAG in production? We tune prefix-cache-aware routing and KV reuse to your real context-reuse pattern, so you get the cache hits the benchmarks promise instead of the evictions that quietly erase them.
• Serving many fine-tunes? We design high-density LoRA serving and a shared KV cache on your Kubernetes and GPU layer, packing adapters without thrashing hardware.
• Chasing cost at scale? We right-size workload-aware autoscaling and GPU partitioning so your accelerators stay busy rather than over-provisioned for peak.
• Bound by data residency, sovereignty, or air-gap rules? We build and operate the whole stack inside your boundary, with the governance and controls your auditors expect.
• Still not sure self-hosting is worth it? That is the most useful conversation to have early, and the cheapest.

Wherever you land on that list, talk to us. We will give you a straight read on whether self-hosting fits before anyone writes a line of YAML.

Sources

• llm-d: v1.0 announcement, CNCF project listing, llm-d.ai
• NVIDIA Dynamo: documentation, developer blog
• AIBrix: vLLM project announcement, documentation
• llm-d and Dynamo interoperation: NVIDIA Technical Blog