AI/LLM Cluster Component Inventory¶

This inventory is used to translate a reference architecture into the actual scope of cluster construction. It does not require installing all projects at once. Instead, first select the target scenario, then install only the components that solve a clearly identified problem.

Components are classified into three levels:

• Required: Without it, the scenario cannot form an operable closed loop;
• Conditionally required: Needed only after reaching multi-tenancy, multi-node, high availability, or a specific data scale;
• Optional: Can be skipped when an equivalent capability already exists or the current scale does not require it.

1. First Select the Target Scenario¶

If you currently have only one model, a small number of GPUs, and one team, prioritize completing Scenario A. Do not install all components of Scenario F directly.

2. Base Components for All Scenarios¶

3. GPU and Node Software Stack¶

Minimum acceptance:

Node Ready
  → GPU Resource allocatable
  → nvidia-smi/vendor tools normal inside Pod
  → Single-GPU matrix computation
  → Multi-GPU Collective
  → Metrics can be associated with Pod, Node, and GPU UUID

Node Ready
  → GPU Resource 可分配
  → Pod 内 nvidia-smi/厂商工具正常
  → 单卡矩阵计算
  → 多卡 Collective
  → 指标能关联 Pod、Node 和 GPU UUID

4. Scheduling, Queues, and Multi-tenancy¶

Kueue, Volcano, and YuniKorn do not all need to be installed. Choose the primary path based on whether you need "admission queue" or "Gang/queue capability replacing the Scheduler," and avoid having the same workload managed by multiple scheduling control planes.

5. Data, Storage, and Model Artifacts¶

It is not recommended to put all data, models, and Checkpoints in one huge RWX PVC, nor to distribute frequently changing hundreds-of-GB weights via container images.

6. Distributed Training Components¶

Trainer, JobSet, and KubeRay are different execution paths and should not all become mandatory user options just for "component completeness." Platforms typically provide one golden path and reserve another for special workloads.

7. Ray Scenario Components¶

See Ray's Role in Large Model Training and Inference for the Ray topic.

8. Big Data on Kubernetes Components¶

Do not put all state and compute into the same node pool just because Spark, Flink, and Kafka can all run on Kubernetes. Kafka/Flink persistent stateful services, Spark elastic batch jobs, and GPU training require different availability and scaling strategies.

9. LLM Inference and Microservice Components¶

See LLM Serving and AI Microservice Frameworks for selection principles. A single service can have only one replica-count owner, one primary routing owner, and one model lifecycle owner.

10. RAG, Agent, and Application Layer¶

Application frameworks do not replace GPU serving. It is recommended that Dify/LangGraph and similar tools call stable model APIs through an internal AI Gateway.

11. Observability and Operations¶

Retain at minimum the following correlation keys: Namespace, Tenant, Queue, Job/Service, Pod, Node, GPU UUID, model version, data Snapshot, image Digest.

12. Security and Governance¶

13. Five Executable Component Packages¶

Package A: Minimal GPU Inference/Fine-Tuning¶

Kubernetes + CNI/CSI
GPU Driver/Device Plugin + DCGM
Registry + Object Storage
Job + Deployment + vLLM
Gateway/Ingress + TLS
Prometheus/Grafana + Logs
GitOps + Secret/Identity

Do not install yet: Kueue, KServe, AIBrix, Ray, LWS, P/D platforms. First prove single-model performance, cold start, release, and failure recovery.

Package B: Multi-Tenant Training¶

Add to Package A:

Kueue (or Volcano/YuniKorn primary path)
Kubeflow Trainer/JobSet (or KubeRay primary path)
RDMA/Network Operator (required for multi-node)
MLflow/Model Registry
Workflow
Shared/parallel storage + Checkpoint
Cost and queue dashboard

Kueue（或 Volcano/YuniKorn 主路径）
Kubeflow Trainer/JobSet（或 KubeRay 主路径）
RDMA/Network Operator（多机需要）
MLflow/Model Registry
Workflow
共享/并行存储 + Checkpoint
成本与队列看板

Package C: Ray Large Model Platform¶

Add to the base layer:

KubeRay Operator
RayJob + Kueue
Ray Data / Train / Tune
RayService / Ray Serve LLM
Object storage + Object Store Spill
Ray/Serve metrics, logs, and controlled access to Dashboard

KubeRay Operator
RayJob + Kueue
Ray Data / Train / Tune
RayService / Ray Serve LLM
对象存储 + Object Store Spill
Ray/Serve 指标、日志和 Dashboard 受控访问

Training and online serving use RayJob and RayService respectively, and do not share the same failure domain.

Package D: Big Data and AI¶

Strimzi/Kafka (real-time)
Flink Operator (stream state)
Spark (batch processing)
Object storage + Iceberg/Delta/Hudi + Catalog
Trino (interactive SQL)
Workflow + data quality/Lineage
CPU/NVMe data node pool
Ray Data or GPU ETL (last mile, optional)

Strimzi/Kafka（实时）
Flink Operator（流状态）
Spark（批处理）
对象存储 + Iceberg/Delta/Hudi + Catalog
Trino（交互 SQL）
Workflow + 数据质量/Lineage
CPU/NVMe 数据节点池
Ray Data 或 GPU ETL（最后一公里，可选）

Package E: Online Inference at Scale¶

Select one control plane on top of Package A based on the target:

Gateway API/Enterprise Gateway
+ KServe or AIBrix or RayService/NIM Operator
+ vLLM/SGLang/NIM Runtime
+ Model node cache
+ KEDA/HPA/dedicated Autoscaler
+ LWS/Kueue (multi-node replicas)
+ llm-d/AIBrix/Dynamo/Ray Serve LLM (only needed for P/D)
+ OTel Trace, Token/KV/Queue metrics

Gateway API/企业 Gateway
+ KServe 或 AIBrix 或 RayService/NIM Operator
+ vLLM/SGLang/NIM Runtime
+ 模型节点缓存
+ KEDA/HPA/专用 Autoscaler
+ LWS/Kueue（多机副本）
+ llm-d/AIBrix/Dynamo/Ray Serve LLM（仅 P/D 需要）
+ OTel Trace、Token/KV/Queue 指标

Do not stack KServe, AIBrix, RayService, and NIM Operator all on the same model service.

14. Recommended Installation Order¶

1. Kubernetes, CNI, CSI, DNS, and node baseline;
2. GPU driver, Device Plugin, Runtime, and DCGM;
3. Registry, object storage, Secret/Identity, and GitOps;
4. Prometheus, logging, alerting, and auditing;
5. Native Job and Deployment + vLLM minimal closed loop;
6. Choose Kueue + training controller, or KubeRay, based on the scenario;
7. Add Spark/Flink/Kafka/Lakehouse based on the data scenario;
8. Choose KServe, AIBrix, Ray Serve, or NIM based on service scale;
9. Add multi-node, P/D, KV Offload, and multi-cluster only when benchmarks prove the need;
10. Add upgrade, backup, failure, and rollback drills with every additional layer.

15. Cluster-Level Acceptance Checklist¶

• Every component corresponds to a clear requirement and Owner;
• No multiple controllers writing to the same resource for the same capability;
• All versions are entered into the Kubernetes, Driver, CUDA, framework, and CRD compatibility matrix;
• GPU baseline for single card, multi-card, cross-node, and faulty-card isolation is complete;
• Jobs can queue, admit, run, save Checkpoints, and be reclaimed;
• Models can cold start, warm up, receive traffic, drain, and roll back;
• Data, models, images, and configuration all use immutable versions;
• Object storage, shared disks, NVMe, and network have completed capacity testing;
• Metrics and logs can be traced from business requests/training runs down to Pod, Node, and GPU;
• Failure drills are complete for the control plane, Operators, Gateway, Head, and state stores;
• Namespace, RBAC, Queue, NetworkPolicy, and credential isolation have passed testing;
• The cluster can be rebuilt from Git, backups, and external artifacts;
• Component upgrades have Canary, rollback, and CRD compatibility plans;
• Can answer "what capability would be lost if this component were removed."

Further Reading¶

• AI/LLM on Kubernetes Reference Architecture
• AI on Kubernetes Implementation Roadmap
• Big Data on Kubernetes
• Ray's Role in Large Model Training and Inference
• LLM Serving and AI Microservice Frameworks
• GPU Platform Minimal Closed-Loop Experiment