Multi-Machine and Disaggregated LLM Inference¶

When the model, KV Cache, or target throughput exceeds the capacity of a single machine, an inference replica grows from one Pod into multiple interdependent Workers. Going further, Prefill and Decode can be split into independent resource pools that transfer the KV Cache over the network.

Such systems improve scaling efficiency, but they also bring networking, topology, rollout, failure, and state management into the request critical path.

1. First Distinguish Three Kinds of Scaling¶

A production system may use all three at once: TP/PP inside each replica, multiple DP replicas, and then split Prefill and Decode Pools.

2. Multi-Machine Model Replicas¶

One logical model replica
  Leader / API Pod
    ├── Worker 0: GPU 0-7, node A
    ├── Worker 1: GPU 0-7, node B
    └── Worker 2: GPU 0-7, node C

一个逻辑模型副本
  Leader / API Pod
    ├── Worker 0：GPU 0-7，节点 A
    ├── Worker 1：GPU 0-7，节点 B
    └── Worker 2：GPU 0-7，节点 C

These need to be managed as a whole:

• Created and deleted together;
• Stable identity and discovery;
• Gang/All-or-Nothing scheduling;
• Networking and topology;
• Readiness of all Ranks;
• Whole-replica recovery when one Worker fails;
• Scaling and rollout in units of complete replicas.

A plain Deployment does not provide these semantics.

2.1 Using DeepSeek-V4-Pro to Understand "Multi-Machine Replica" vs. "Multiple Replicas"¶

DeepSeek-V4-Pro is well suited to illustrate these two easily confused levels. The official public specification is 1.6T total parameters, 49B activated parameters, 1M Context; the model weights use FP4 and FP8 mixed precision. The official vLLM recipe estimates the mixed-precision checkpoint at about 960GB and gives a concrete multi-node example: one GB200 NVL4 Tray has 4 GPUs, and a single Tray cannot hold this checkpoint, so 2 Trays with 8 GPUs total are used, sharing the model via DP + EP.

References: DeepSeek-V4 official release, DeepSeek-V4 open weights, vLLM DeepSeek-V4-Pro Recipe

These 2 nodes and 8 GPUs are, at the platform layer, first of all one complete model replica:

DeepSeek-V4-Pro Replica A
  Leader / API Endpoint
    ├── GB200 NVL4 Tray A1: 4 GPUs
    └── GB200 NVL4 Tray A2: 4 GPUs
        └── vLLM DP + Expert Parallel communication domain

DeepSeek-V4-Pro Replica A
  Leader / API Endpoint
    ├── GB200 NVL4 Tray A1：4 GPU
    └── GB200 NVL4 Tray A2：4 GPU
        └── vLLM DP + Expert Parallel 通信域

The scheduler must prepare nodes, GPUs, networking, and model data for the entire group at once; if any required Rank is not Ready, Replica A must not enter the external Endpoint. Higress or AIBrix must also not send user requests directly to an individual Worker Pod; they may only access this group's Leader/API Service.

When an enterprise deploys another set of identical resources for throughput and availability, "multiple complete replicas" appear:

Logical model deepseek-v4-pro
  ├── Replica A: 2 trays / 8 GPUs
  └── Replica B: 2 trays / 8 GPUs

逻辑模型 deepseek-v4-pro
  ├── Replica A：2 Tray / 8 GPU
  └── Replica B：2 Tray / 8 GPU

There are now two different scheduling decisions:

1. Kubernetes, Kueue, and StormService/RoleSet, LeaderWorkerSet, or KubeRay decide how each 8-GPU group lands on nodes as a whole;
2. The AIBrix Gateway decides whether a given model=deepseek-v4-pro request goes to Replica A or B, and must not mix the internal Workers of A/B into a plain Endpoint pool.

If only Replica A exists, a stable Leader Service is sufficient to carry traffic, and AIBrix's "replica selection" adds limited value; only with multiple groups A/B, P/D pools of both types, significant queue differences caused by long contexts, or a desire for KV/Session awareness does internal inference routing become a necessary capability.

Hardware numbers cannot be copied directly. The same vLLM recipe also gives different parameters for 8×B300, 8×H200, and other platforms; the actual node count depends on GPU memory, weight precision, the KV Cache produced by Context/concurrency, Runtime version, and parallelism strategy. Before going live, you must re-measure against the target hardware rather than deriving the node count directly from "1.6T parameters."

2.2 Besides DeepSeek-V4, What Existing Examples Are There¶

First distinguish three kinds of "support," otherwise a model list can easily be mistaken for production cases that have actually been run:

1. AIBrix official end-to-end examples: the repository contains orchestration, label, Service, and routing manifests;
2. vLLM official multi-machine recipes: the model and multi-machine parallelism strategy have upstream basis, but the Runtime still needs to be integrated with AIBrix;
3. Theoretically OpenAI API compatible: this only shows that AIBrix can discover and forward; it does not prove that multi-machine communication, P/D, or scaling have been verified.

As of August 2026, the three example sets in the AIBrix repository most worth referencing directly are:

Real large-model candidates are not limited to DeepSeek. The models in the table below are all explicitly declared in the vLLM official Recipes to be compatible with multi_node_tp, multi_node_tp_pp, distributed Expert Parallel, and pd_cluster, so they are suitable for connecting the vLLM Leader/API Endpoint to AIBrix:

The GPU memory in the table is only the minimum order of magnitude the recipe gives for the weight variant; it does not include the full KV Cache, CUDA Graph, communication buffers, and safety margin required for the target concurrency, nor does it equal a recommended GPU count. In particular, NVFP4 usually requires native Blackwell capability; you cannot simply add up total GPU memory.

2.3 The Correct Boundary for Integrating a Model into AIBrix¶

AIBrix does not have a whitelist that only permits specific models. For an ordinary colocated vLLM service, at minimum you must ensure:

• Pods carry consistent model.aibrix.ai/name and model.aibrix.ai/port;
• vLLM's --served-model-name, request body model, Service, and routing configuration are consistent;
• The Service selects only the Leader/API Pod of one complete replica and must not select internal Headless Workers;
• Readiness succeeds only after all required Ranks and the model are servable;
• The AIBrix Gateway routes between multiple complete replicas, while intra-group Ranks are managed by RayClusterFleet, StormService/RoleSet, LWS, or the Runtime.

In non-P/D scenarios, the coupling between the AIBrix Gateway and the vLLM engine version is relatively weak; as long as a newer vLLM keeps compatible APIs, metrics, and discovery labels, it can usually be integrated first as an ordinary model Endpoint. P/D, KV Offload, NIXL Connector, and Runtime Sidecar sit on the data path, where version coupling is strong. Recipes such as Qwen3.5, Kimi-K2.5, and MiniMax-M2.5 require vLLM 0.17—0.20, which is clearly newer than the AIBrix v0.7.0 example images; you must rebuild images and validate the KV Connector, NIXL, CUDA/NCCL, and metrics, rather than only replacing the model field.

Recommended rollout in increasing risk order:

1. First use the Qwen2.5-Coder-7B two-node example to prove RayClusterFleet and Head-only routing;
2. Use Qwen3-8B to prove the StormService and P/D control chain, but do not yet claim performance benefits from it;
3. For the first real large model, prefer Qwen3-235B-A22B FP8, or DeepSeek-R1 671B, which has more complete reproduction evidence;
4. Then add a second complete replica to validate AIBrix queueing, Prefix/Session awareness, failure removal, and scaling;
5. Only last test 1T models such as Kimi-K2.5 and cross-node P/D, and decide whether to go live based on TTFT, TPOT, Goodput, and failure recovery time.

2.4 AIBrix's Two Orchestration Routes¶

AIBrix can use RayClusterFleet to manage multiple complete RayCluster replicas, or StormService → RoleSet → Pod to directly express Prefill, Decode, or other roles. RoleSet is not a third option on the same level as the previous two, but the next-level resource managed by StormService.

This section only establishes the conceptual boundary: when Ray is already in use, RayClusterFleet is usually preferred; when P/D separation, role-level elasticity, or removing Ray is needed, StormService is usually preferred. A complete comparison of CRD hierarchy, dependencies, Endpoints, scaling, updates, and failure boundaries, as well as alternative approaches on bare Kubernetes, is consolidated in Orchestration Option Selection in AIBrix Practice.

3. LeaderWorkerSet¶

LeaderWorkerSet (LWS) describes a replication unit consisting of one Leader and multiple Workers, suitable for multi-machine inference and other Leader/Worker workloads.

It provides:

• Group-level replicas;
• Stable Leader/Worker indices and discovery;
• Ordered/parallel lifecycle capabilities;
• Group-level rolling updates and scaling;
• Integration with Kueue, Gateway, and the inference control plane.

LWS does not handle model request routing, quota admission, or KV Cache. Typically you also need:

• Kueue for group admission and topology;
• Gateway/InferencePool for request selection;
• The engine to handle TP/PP communication;
• Model cache and RDMA components for the data path.

Reference: LeaderWorkerSet

4. Topology Selection¶

The usual priority order is:

1. Keep TP within the NVLink/NVSwitch domain whenever possible;
2. When PP crosses nodes, ensure stable RDMA/high-speed networking;
3. For MoE EP, pay attention to All-to-All and Expert hotspots;
4. Keep CPU, GPU, and NIC in reasonable NUMA/PCIe proximity;
5. Spread multiple replicas across failure domains, while concentrating within a single replica;
6. Place the model cache on the selected nodes.

"Concentrate within a replica" and "spread between replicas" are two different goals that require a layered topology strategy.

5. vLLM Multi-Machine Parallelism¶

vLLM supports Tensor/Pipeline and other distributed inference. Typical approaches:

• Within a single node, TP equals the number of GPUs used on that machine;
• When the model crosses nodes, add PP;
• Multi-replica throughput is achieved via DP or multiple upper-layer Deployments/LWS;
• The specific Runtime can use Ray or another supported execution backend.

Illustration:

vllm serve /models/example \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2

This represents a conceptual configuration where one logical replica uses 16 GPUs; in practice, multi-node startup, address discovery, container shared memory, and network parameters are also required.

Currently, vLLM multi-node deployments can use either Ray or the native MultiProcessing modes --nnodes, --node-rank, and --headless. Ray is a distributed execution backend, not the only parallel algorithm for multi-machine inference itself; TP, PP, DP, and EP determine how model computation is split.

Reference: vLLM Parallelism and Scaling

5.1 What is the relationship between vLLM Ray multi-node and P/D disaggregation¶

They are two orthogonal dimensions:

• vLLM Ray multi-node answers: how a single vLLM Engine splits the computation of the same model replica across multiple GPUs and multiple nodes;
• P/D disaggregation answers: whether the Prefill and Decode of a single request are handled by two independent vLLM Engines, and how the KV Cache is transferred from the former to the latter.

The most important insight is: Ray Workers are not Decode Workers. In ordinary Ray multi-node mode, the Head/Worker Ranks jointly participate in the Prefill and Decode of the same Engine; in P/D mode, Prefill and Decode are two Engines with independent Schedulers, Batches, KV Caches, and Endpoints.

5.1.1 Ray only, without P/D¶

Client
  → vLLM API / Ray Head
      → Same vLLM Engine
          ├─ Ray Rank 0: GPU/weight shard
          ├─ Ray Rank 1: GPU/weight shard
          └─ Ray Rank 2: GPU/weight shard
             All ranks jointly execute Prefill and Decode

Client
  → vLLM API / Ray Head
      → 同一个 vLLM Engine
          ├─ Ray Rank 0：GPU/权重分片
          ├─ Ray Rank 1：GPU/权重分片
          └─ Ray Rank 2：GPU/权重分片
             所有 Rank 共同执行 Prefill 和 Decode

Here the model is placed across GPUs via TP/PP/EP. A request belongs to the same Engine from start to finish; Ranks continuously perform Collective, Activation, or Pipeline communication during layer, token, or expert computation. Ray handles process placement and execution coordination, while vLLM still handles the inference Scheduler.

5.1.2 P/D only, without Ray¶

Client
  → P/D Router
      → Prefill vLLM Instance: load model, compute prompt and initial KV
          → Connectors such as NIXL/LMCache/Mooncake transfer KV
      → Decode vLLM Instance: also loads model, receives KV and generates token by token

Client
  → P/D Router
      → Prefill vLLM Instance：加载模型，计算 Prompt 和初始 KV
          → NIXL/LMCache/Mooncake 等 Connector 传输 KV
      → Decode vLLM Instance：也加载模型，接收 KV 后逐 Token 生成

Each side can use just a single GPU, or single-node multi-GPU MultiProcessing. P/D does not require Ray, but it does require a P/D-aware Router, KV Connector, Peer discovery, and compatible models, KV layouts, and Engine versions. Neither Prefill nor Decode is "only half a model installed": both sides must have the full logical model capability required to execute their respective stages, so this typically results in two copies of weight capacity, though each copy can still be sharded internally.

5.1.3 Ray and P/D can be combined¶

Client
  → P/D Router
      → Prefill Engine (RayCluster P)
          ├─ P-Head
          └─ P-Worker × N
              → Transfer KV across Engines
      → Decode Engine (RayCluster D)
          ├─ D-Head
          └─ D-Worker × M

Client
  → P/D Router
      → Prefill Engine（RayCluster P）
          ├─ P-Head
          └─ P-Worker × N
              → 跨 Engine 传输 KV
      → Decode Engine（RayCluster D）
          ├─ D-Head
          └─ D-Worker × M

In this case, Ray handles model parallelism within P and D respectively, while the P/D Router and Connector handle request switching and KV handoff between the two Engines. Prefill and Decode can use different TP/PP shapes, but this configuration is only possible when the KV Connector's TP Mapping and compatibility matrix explicitly support it; arbitrary combinations are not allowed.

The official AIBrix v0.7.0 examples separately demonstrate RayClusterFleet multi-node and StormService P/D; they are not yet a ready-to-apply "RayClusterFleet nesting StormService" manifest. Enterprises that need the combined architecture above should treat it as an advanced integration: manage the P/D distributed Engines separately, expose only their respective Leader/API Endpoints to the Router, and independently verify discovery, pairing, KV transfer, and failure recovery.

5.1.4 Comparison of similarities and differences¶

Four common combinations for quick architecture assessment:

Do not introduce P/D simply because Ray is already in use. The current vLLM official documentation explicitly describes the primary value of P/D as separately tuning TTFT/ITL and controlling tail ITL, and reminds users that it does not automatically improve raw throughput. The platform layer may achieve higher SLO Goodput or utilization through independent scaling ratios, but the benefits must cover the costs of KV transfer, duplicate weights, an additional Router, and the expanded failure surface.

References: vLLM Disaggregated Prefilling, NixlConnector Usage Guide, NixlConnector Compatibility Matrix

6. Why Prefill and Decode are different¶

In colocated mode, the same set of GPUs shares resources between the two types of tasks. When load is unbalanced, the following issues can occur:

• Long Prompts block interactive Decode;
• Prefill peaks and Decode capacity cannot be scaled separately;
• Different hardware cannot each leverage its own strengths;
• Queues and Batches interfere with each other.

7. P/D disaggregation architecture¶

Client
  → Router / Frontend
      → Prefill Pool
          → Generate KV Cache
          → Transfer via NIXL/UCX/RDMA/other Connectors
      → Decode Pool
          → Continue generation and stream back

Client
  → Router / Frontend
      → Prefill Pool
          → 生成 KV Cache
          → 通过 NIXL/UCX/RDMA/其他 Connector 传输
      → Decode Pool
          → 继续生成并流式返回

Sources of benefit:

• Prefill and Decode scale independently;
• Different GPU models or parallel configurations can be used;
• Batch, concurrency, and SLO can be tuned separately;
• Better isolation of long-Prompt impact on Decode;
• Higher utilization and Goodput at sufficient scale.

New costs:

• KV transfer latency and bandwidth;
• Peer discovery and connection management;
• Dual-pool capacity balancing;
• Cancellation, failure, and orphan states;
• Release, rollback, and version compatibility;
• More components and observability boundaries.

8. When P/D disaggregation should NOT be used¶

• A single machine or a small number of GPUs already satisfies the SLO;
• Requests are short and KV transfer cost approaches or exceeds the benefit;
• The network does not have stable high-bandwidth, low-latency capability;
• The Prefill/Decode traffic ratio is stable and colocated efficiency is already good;
• The team does not yet have complete benchmarks, traces, and failure handling;
• The model/Engine/Connector combination does not have production support;
• Cold-start and release complexity matters more than GPU savings.

P/D disaggregation is a scale optimization, not a prerequisite for deploying LLMs.

9. KV transfer path¶

The following must be clarified:

• KV format, Block size, Dtype, and layout;
• Whether Prefill/Decode Engines and versions are exactly identical;
• Who initiates the transfer, Push or Pull;
• How Peers are discovered and authenticated;
• Whether GPU Direct, RDMA, TCP, or shared storage is used;
• When the KV is released on the sending side;
• How cleanup is handled after Decode failure;
• How request cancellation propagates;
• Whether transfer timeouts and retries are safe;
• Whether the KV contains sensitive Prompt information.

A single model or Engine upgrade can change the KV layout; old and new Workers must not be arbitrarily interconnected.

10. llm-d¶

llm-d is a Kubernetes-oriented distributed LLM inference architecture built around:

• InferencePool and Endpoint Picker;
• Model serving with vLLM/SGLang and others;
• Prefix/KV-aware Routing;
• KV Cache Indexing and Offload;
• P/D disaggregation;
• Flow Control;
• Kubernetes Deployment, Prometheus, and Gateway integration.

It is suitable for teams that want to build a composable inference data plane using Kubernetes-native APIs. For production selection, pin the llm-d, Gateway Extension, Engine, Connector, and CRD versions.

Reference: llm-d Architecture

11. NVIDIA Dynamo¶

Dynamo provides Frontend, Router, Planner, Worker, and disaggregated inference capabilities. It can compose vLLM, SGLang, TensorRT-LLM, and other Backends, and transfers KV via mechanisms such as NIXL.

Suitable for:

• Large-scale distributed inference on NVIDIA platforms;
• P/D, KV Routing, and Expert Parallel that have been validated through recipes;
• Expressing component graphs via resources such as DynamoGraphDeployment;
• Topology and capacity planning before deployment.

Dependencies such as etcd/NATS for discovery or messaging, Dynamo CRD/Operator, Runtime images, and Backend compatibility must be managed.

Reference: NVIDIA Dynamo Disaggregated Serving

12. What alternatives exist beyond AIBrix¶

First, distinguish three layers to avoid treating inference Engines, Kubernetes orchestration components, and complete inference control planes as comparable products:

Platform control plane: AIBrix / llm-d / KServe / Ray Serve / Dynamo
Cluster orchestration layer: Kubernetes / RBG / LeaderWorkerSet / KubeRay
Inference execution layer: vLLM / SGLang / TensorRT-LLM

平台控制面：AIBrix / llm-d / KServe / Ray Serve / Dynamo
集群编排层：Kubernetes / RBG / LeaderWorkerSet / KubeRay
推理执行层：vLLM / SGLang / TensorRT-LLM

A platform often uses components from all three layers simultaneously. For example, KServe handles service lifecycle, LeaderWorkerSet expresses cross-node replicas, and vLLM performs model computation; they are not three interchangeable products.

12.1 Selection conclusions¶

• Looking for the open-source option closest to AIBrix's positioning: prioritize evaluating llm-d;
• Already have a KServe platform: use KServe LLMInferenceService + LeaderWorkerSet;
• NVIDIA GPUs, RDMA, and topology conditions are mature: test Dynamo + TensorRT-LLM/vLLM;
• Already using Ray extensively for data processing, training, and inference: choose KubeRay + Ray Serve LLM;
• Want to productionize vLLM first without rushing to build a unified control plane: start with vLLM Production Stack;
• Need a multi-role lifecycle API independent of the Gateway: evaluate RBG; if the full AIBrix stack is already adopted, compare StormService first to avoid two controllers owning the same set of Pods;
• Need a self-developed enterprise inference platform: compose Gateway API + llm-d Router/EPP + KServe/LWS + vLLM/SGLang, but must ensure each resource has exactly one lifecycle and scaling controller.

References: RBG, llm-d Architecture, KServe LLMInferenceService, Ray Serve LLM, vLLM Production Stack, SGLang. For the detailed boundary between RBG and AIBrix/Higress and the sr1 measurements, see RBG Multi-Role Inference Orchestration; for the SGLang Router CPU control chain measurements, see SGLang Model Gateway CPU in Practice.

13. Dual-Pool Capacity Planning¶

Measure separately:

Prefill demand ≈ input token arrival rate / effective tokens per second of a single Prefill Worker
Decode demand ≈ output token arrival rate / effective tokens per second of a single Decode Worker

Prefill 需求 ≈ 输入 Token 到达率 / 单 Prefill Worker 的有效 Token/s
Decode 需求  ≈ 输出 Token 到达率 / 单 Decode Worker 的有效 Token/s

Also include:

• Long and short prompt distribution;
• KV transfer concurrency and bandwidth;
• Single Worker failure redundancy;
• Peak and target utilization;
• Model/Cache warm-up;
• Rejection and queueing budget.

The Autoscalers of the two Pools cannot only look at their own local queues. If Prefill scales up too quickly, it will overwhelm Decode; if Decode is over-provisioned, it will idle waiting for KV.

14. Release and Versioning¶

A disaggregated release includes at least:

• Model artifact;
• Prefill runtime and parameters;
• Decode runtime and parameters;
• KV Connector/NIXL/UCX versions;
• Router/EPP;
• Gateway and CRD;
• GPU driver, CUDA/ROCm;
• Network and RDMA configuration.

The safest Canary approach is to stand up an entirely new Pool, rather than letting new Prefill workers randomly connect to old Decode workers.

Release process:

1. Create a new-version Prefill/Decode Pool;
2. Pass compatibility and KV transfer tests;
3. Warm up models and connections;
4. Shadow side-effect-free traffic;
5. Canary by tenant/request in a stable manner;
6. Compare quality, TTFT, TPOT, Goodput, and cost;
7. Stop new requests to the old Pool;
8. Reclaim after draining.

15. Failure Semantics¶

16. Observability¶

It must be possible to see from a single request:

Gateway
  → Router selection reason
  → Prefill queueing/execution
  → KV size, transfer time and path
  → Decode queueing/execution
  → Token streaming return

Gateway
  → Router 选择原因
  → Prefill 排队/执行
  → KV 大小、传输时间和路径
  → Decode 排队/执行
  → Token 流式返回

Metrics:

• Prefill/Decode respective queues, batch, and tokens/s;
• KV transfer bytes, time, failures, and retries;
• Peer count, connection establishment and teardown;
• TTFT breakdown into routing, queueing, Prefill, and transfer;
• TPOT and Decode batch;
• Orphaned KV, cancellations, and cleanup;
• Pool scaling, Ready status, and version;
• RDMA/NIC/GPU/CPU metrics.

17. Security¶

• KV Cache may contain reconstructable user context; treat it as sensitive data;
• Use network identity, encryption, or a controlled data plane between Prefill and Decode;
• Access boundaries for KV Index and Cache across different tenants are clearly defined;
• Workers are not exposed to the public network;
• The Router must not send requests to unauthorized models/Adapters;
• Debug logs do not record raw KV, prompts, or credentials;
• Multi-NIC/RDMA ports are protected through network isolation and node policies.

18. Go-Live Checklist¶

• It has been demonstrated that single-machine/colocated mode cannot meet the goals more simply.
• A multi-machine replica is created, scheduled, and released as a whole by LWS/controller.
• Replicas are concentrated internally and spread across failure domains between replicas.
• KV format, Connector, engine, and model versions are fully locked down.
• The P/D two Pools use a joint capacity model and protection mechanisms.
• Request cancellation, Worker failure, and KV timeout have been exercised.
• Releases do not let new and old incompatible Workers pair randomly.
• Traces can break down routing, Prefill, KV transfer, and Decode.
• Safe degradation is possible when Cache Index/Router fails.
• KV Cache privacy, network, and tenant isolation have been reviewed.

Further Reading¶

• LeaderWorkerSet
• llm-d
• llm-d Disaggregated Serving
• NVIDIA Dynamo Disaggregated Serving
• TensorRT-LLM Disaggregated Serving
• KServe LLMInferenceService

For multi-region scenarios, prefer deploying a complete inference stack in each cluster, with a global Gateway selecting the cluster and the in-cluster Router/EPP selecting the model replica; TP/PP, P/D, and KV transfer should in principle stay within a low-latency network domain. See: Kubernetes Cross-Cluster and Large-Scale GPU.