Skip to content

AI/LLM on Kubernetes Infrastructure

This documentation is not about "how to get a GPU Pod running", but about how to build Kubernetes into a sustainable, evolving AI/LLM infrastructure: devices can be discovered and isolated, jobs can queue fairly, data can keep up with compute, training can recover, inference can meet latency targets, models and dependencies can be traced, and the platform can be upgraded, observed and audited.

The content covers the complete chain from hardware to applications, and attempts to answer four types of questions: what each component is responsible for, when it is needed, how to verify it, and where to troubleshoot when problems arise.

Browse by Topic View Reference Architecture Run GPU Platform Experiments Go to Hands-On and Troubleshooting

Infrastructure Overview

English rendering — the original text is retained below.

Users, SDKs, applications, and workflows
        │
        ├─ Big data / RAG / Agent / Online inference / Training tasks
        │
Gateway, Serving, Trainer, Pipeline, Queue
        │
Kubernetes API, controllers, scheduler, admission and policies
        │
Device Plugin / CDI / DRA / CNI / CSI / CRI
        │
GPU, TPU, NPU, CPU, RDMA, NVMe, object storage
        │
Power, cooling, racks, failure domains, and data center network
Original text (verbatim) 
用户、SDK、应用与工作流
        │
        ├─ 大数据 / RAG / Agent / 在线推理 / 训练任务
        │
Gateway、Serving、Trainer、Pipeline、Queue
        │
Kubernetes API、控制器、调度器、准入与策略
        │
Device Plugin / CDI / DRA / CNI / CSI / CRI
        │
GPU、TPU、NPU、CPU、RDMA、NVMe、对象存储
        │
供电、散热、机架、故障域与数据中心网络

Any layer can become the bottleneck. Low GPU utilization is not necessarily a GPU problem; it may be caused by queuing policy, CPU decoding, model download, storage throughput, network congestion, request routing, or batching parameters. When reading and troubleshooting, you should verify layer by layer along the complete request or job path.

Where to Start

Your Goal Suggested Path
Understand from scratch how AI runs on Kubernetes How Kubernetes Hosts AI → Cluster Architecture Design → Glossary
Build or take over a GPU cluster GPU Node Software Stack → Device Management → GPU Scheduling → Platform Operations
Choose an open-source cluster management platform Open-Source Cluster Management Tools and Approaches → Platform Operations → Cross-Cluster and Large-Scale GPU
Choose domestic or international cloud vendor Kubernetes Cloud Vendor Managed Kubernetes → Cluster Architecture Design → Heterogeneous Accelerators
Build a GPU Notebook development platform GPU Notebook Platform and Storage → GPU Scheduling → Data and Caching → MLOps
Build a big data and AI data platform Big Data on Kubernetes → Data and Caching → Model Artifacts → MLOps
Build a multi-tenant training platform Queues and Multi-Tenancy → Distributed Training → RDMA Networking → Reliability
Build a Ray large model platform Ray Training and Inference → Distributed Training → Big Data on Kubernetes → Reliability
Build online LLM inference services Local Running and Testing → Inference Platform Overview → Inference Engines → Serving Frameworks → API Relay and Gateway Selection → Gateway and Routing → Higress in Practice
Plan multi-region or multi-GPU clusters Cluster Architecture Design → Cross-Cluster and Large-Scale GPU → Production Reference Architecture
Build a RAG or Agent platform Agent Status and Trends → Agent Harness → HarnessRouter → DeepSeek Harness Containerization → Kubernetes in Practice → Agent Sandbox Selection → Security Governance
Responsible for SRE, cost or capacity Observability → Performance Benchmarks → Cost and Capacity → Adoption Roadmap

Complete Topic Map

Foundations and Architecture

Clusters, Nodes and Accelerators

Queues, Elasticity and Capacity

Networking, Data and Model Artifacts

Distributed Training

LLM Inference

RAG, Agent and Edge

Platform Engineering and Production Operations

Reference Architectures and Experiments

Tools Are Not Architecture

Multiple projects are usually available for the same layer, and projects may overlap. Before choosing a tool, write down the inputs, outputs, owners, and acceptance metrics first.

Capability Common Implementations Questions to Answer First
Device access Device Plugin, CDI, DRA, vendor Operators Do you need full-GPU, slicing, topology, or dynamic claims?
Batch scheduling kube-scheduler, Kueue, Volcano Do you need admission queues, fair sharing, or Gang Scheduling?
Big data compute Spark, Flink, Trino, Ray Data Is it batch processing, streaming state, interactive SQL, or AI data preparation?
Training control Kubeflow Trainer, JobSet, KubeRay What are the training framework, fault tolerance, and elasticity boundaries?
Model serving KServe, Seldon, custom controllers Do you need standard model APIs, multi-model, or LLM-specific capabilities?
Inference runtime vLLM, SGLang, TensorRT-LLM, Triton What are the target model, hardware, latency, and throughput?
Request ingress Gateway API, Inference Extension, service mesh Does routing need to understand model, KV Cache, and queue state?
Agent execution Agent Sandbox, gVisor, Kata, managed Sandbox APIs Are lifecycle, isolation runtime, and tool authorization already layered?
Observability Prometheus, OpenTelemetry, DCGM Exporter Can you trace from user requests down to Pod, GPU, network, and model version?

Content Status and Update Rules

  • stable: Principles and production methods are relatively stable, but still need to be validated against your target Kubernetes and component versions.
  • evolving: Upstream APIs, project boundaries, or best practices are still changing rapidly; check official release notes before going live.
  • lab: Minimal experiments for reproduction and learning; should not be copied directly into production without review.
  • last_reviewed: Date of the most recent review against primary sources; does not mean all dependencies are pinned to that date.
  • Examples primarily express structure and validation methods; production configurations must additionally complete image pinning, capacity, identity, secrets, backup, SLOs, and change processes.
  1. First establish a repeatable acceptance baseline for nodes, drivers, networking, storage, and GPU.
  2. Then unify the resource model, queues, tenants, images, model artifacts, and observability.
  3. Establish golden paths for training, online inference, offline inference, RAG, and agents respectively.
  4. Use performance benchmarks, failure drills, cost metrics, and upgrade matrices to form continuous delivery gates.
  5. Finally, introduce DRA, advanced topology, disaggregated inference, multi-cluster, and complex scheduling policies based on scale.

If you are planning a first-version platform, you can start from the Production Reference Architectures and pick the closest starting point, then use the Adoption Roadmap to break it into phased goals.