Skip to content

AI/LLM on Kubernetes Infrastructure Knowledge Base

简体中文 English

Hi~Hello👋, this is runzhliu's work notebook. I am an engineer with ten years of experience in AI/LLM/Big Data/Kubernetes, and I am also a note-taking fanatic🤔 who particularly loves taking notes. This site is a collection of notes summarized from my work experience in AI/LLM Infras over the past few years, polished with the help of large language models. I hope to exchange ideas and learn together with everyone, and do a little🤏 something in the era of large models. Whether it's about the content itself or work, feel free to reach out to me😄👏! (runzhliu@163.com)

This is a set of engineering documentation for platform engineers, SREs, data platform, training, and model serving teams, systematically organizing the infrastructure knowledge required to run AI/LLM workloads on Kubernetes.

Content starts with GPU nodes, heterogeneous devices, big data, scheduling queues, RDMA, and data provisioning, then extends to distributed training, LLM inference, RAG, Agent sandboxes, observability, security, cost, and production operations. Each topic strives to explain component boundaries, selection criteria, key metrics, failure paths, and rollout checklists, rather than merely listing project names.

Enter the Full Technical Map View Production Reference Architecture Hands-on Practice & Troubleshooting Browse Engineering Case Studies RSS Subscription

Commonly Used Official Websites

Direct entry points to official documentation organized by technical layer. Project capabilities, version compatibility, and installation methods change quickly; always refer to the official documentation and the target version's Release Notes when in use.

Documentation Coverage

  • Clusters & Accelerators

    Understand the complete path of AI workloads from the Kubernetes API to node drivers, container runtimes, and GPUs, establishing repeatable node acceptance and heterogeneous device management practices.

    Start with the Infrastructure

  • Big Data, Scheduling & Distributed Training

    Handle Spark/Flink/Kafka/Lakehouse, resource declarations, queue admission, Gang Scheduling, RDMA, data loading, Checkpoint, and failure recovery.

    View Big Data Infrastructure

  • LLM Inference & RAG

    Cover inference engines, batching, KV Cache, quantization, intelligent routing, Prefill/Decode disaggregation, as well as vector retrieval and permission filtering.

    View the Inference Technical Map

  • Platform Engineering & Production Operations

    Keep the platform running long-term with version matrices, benchmarks, SLOs, observability, security policies, capacity models, and release gates.

    View the Platform Operations System

Start by Your Task

Current Task Recommended Reading Path
First systematic understanding of AI on Kubernetes How Kubernetes Carries AI → Cluster Architecture Design → Full Technical Map
Building a GPU or heterogeneous compute cluster GPU Node Software Stack → Device Management → GPU Scheduling
Building a GPU Notebook development platform Notebook Platform & Storage → Data & Caching → MLOps
Building a big data and AI data platform Big Data on Kubernetes → Data & Caching → Model Artifacts → MLOps
Building a multi-tenant training platform Queues & Multi-tenancy → Distributed Training → RDMA Networking → Reliability
Building a Ray large model platform Ray Training & Inference → Distributed Training → Reliability
Building an LLM online inference platform Inference Platform Overview → Engine Selection → Serving Frameworks → API Relay & Gateway Selection → Intelligent Routing → Higress Hands-on
Building RAG or Agent services Agent Current State & Trends → Agent Harness → HarnessRouter → RAG Infrastructure → Agent Sandbox Selection → Security Governance
Responsible for stability, cost, and capacity Observability → 10k-Node Cluster Optimization → API Server Load Balancing → Multi-cluster Stability → Cost & Capacity

Core Topics

Compute & Clusters

Scheduling, Networking & Data

Training & Inference

Application Infrastructure & Governance

Ready-to-Use Reference Materials

Practical Cases

Engineering Cases separately collects executable checklists and real-environment records. Among them, AI/LLM Cluster Component Checklist is used to determine the scope of construction, and K3s Cluster Upgrade and DRA Preflight records a real-environment version span assessment, backup, compatibility check, upgrade, and verification process.

Content Standards

  • Prioritize primary sources such as Kubernetes SIGs, official project documentation, and public specifications.
  • Distinguish between stable principles, rapidly evolving upstream capabilities, and experimental configurations used only for learning.
  • Selection conclusions must be tied to throughput, latency, reliability, cost, security, or operational complexity.
  • Examples must demonstrate verification methods; production deployment must also cover version pinning, permissions, secrets, capacity, backup, and rollback.
  • Preserve existing page URLs; organize new content by topic directory to avoid broken links when navigation changes.

It is recommended to start with the Complete Technical Map, then choose a reading path based on your current role.

Follow the WeChat Public Account

More hands-on AI/LLM on Kubernetes practices, deployment records, and performance analysis are published on the WeChat public account AI-K8S技术工程. Scan the QR code with WeChat, or search for AI-K8S技术工程 in "WeChat Search":

Scan the QR code or follow the public account AI-K8S技术工程 via WeChat Search