Practical Experience, Troubleshooting, and Selection¶
This series no longer explains "what it is" by component, but starts from a specific problem and provides evidence chains, experimental variables, acceptance criteria, and stopping conditions. The numbers on the page should be measured on target hardware and real workloads; example values cannot replace production baselines.
Troubleshooting and Incidents¶
- Why is a Pod still Pending when GPU is idle
- Why is GPU utilization so low
- GPU Node Failure Atlas
- Seemingly NCCL failure, actually Host Memory OOM
- AI Cluster Incident Retrospective Methods
- Stability Building for Hundreds of Kubernetes Clusters
- Practical Optimization of a 10,000-Node Kubernetes Cluster: Control Plane, etcd, Scheduling, and Networking
- Kube-apiserver Load Balancing in Practice: HTTP/2, TLS Termination, and Layer 4/Layer 7 Selection
- Contiv, OVS, and Cilium Chained Networking in Practice: Why kube-proxy Might Not Receive Packets
- From Contiv to Kube-OVN: A Single-Node Migration Keeping the Physical VLAN
- Koordinator Webhook Multi-Replica Certificate Race in Practice: From 60 409s per Second to Shared CA Convergence
- APF Isolation in Practice: Why Zero-Seat Reject Still Allows Requests
Performance, Cost, and Capacity¶
- DeepSeek-V4.1-Flash-1017 Day0: vLLM on Eight H20 GPUs and Million-Token Tests
- vLLM Sleep Single-GPU Dual-Model Test: Recovery Time, GPU Memory, and DRA Design
- MiMo-V2.6-Pro-RL First Test: vLLM Deployment and Performance on Eight H20-3e GPUs
- Qwen-Image-2.1: SGLang and vLLM-Omni Test on a Single L20 GPU
- Qwen3.8-2.4T-A95B-FP8 Day 0: SGLang Deployment and Stress Test on 32×H20-3e
- MiniMax-M3: SGLang and vLLM Test on Eight H20 GPUs
- MiniMax H3 H20-3e Test: Full Audio/Video Generation, Dual Engine, and Persistence Acceptance
- DeepSeek-V4-Flash-Vision-Exp Day 0: Multimodal Deployment and Stress Test on 4×H20
- GLM-5.3 First-Day Test: SGLang and vLLM Baseline Stress Test on 8×H20
- Hy4-preview BF16 Dual-Node H20 Test: How to Choose Between SGLang and vLLM
- Ling-3.0-flash BF16 Single-Node 4×H20 Test: How to Choose Between SGLang, vLLM, and MTP
- Qwen3.8-Flash-Next Day 0 Practice: Running Native 262K on 4×H20
- SGLang v0.5.16 / v0.5.17 / v0.5.18 Single L20 Test: Is the Upgrade Really Faster?
- DeepSeek V4 Flash Distributed KV Cache: From P/D Direct Transfer to Global Cache Pool
- From Half an Hour to Five Minutes: Full-Chain Optimization of Large Model Cold Start
- Large Model Weight Distribution and Loading Acceleration: Community and Commercial Solution Selection
- vLLM, SGLang, and TensorRT-LLM Test on the Same Machine
- Model GPU Memory and Concurrency Capacity Calculator
- Performance Inflection Point of Prefill/Decode Separation
- GPU Cloud Resource Selection at Home and Abroad
Platform and Scheduling¶
- NVIDIA NVCRE: GPU Cluster Readiness and Production Integration
- Large-Scale Kubernetes Monitoring Solutions: Prometheus, VictoriaMetrics, Thanos, and Mimir
- Large-Scale Kubernetes Scheduler Optimization: Queues, Plugins, Search, and Parallel Architecture
- OpenTelemetry × Grafana: Observing Kubernetes Control Plane and Cluster State
- Pitfalls of GPU Development Platforms in the Large Model Era
- For a 4-GPU Task on an 8-GPU Node, How to Avoid Getting Fragmented GPUs Across NUMA
- GPU Resource Bank and Tidal Inference Platform Practice Blueprint
- Kueue vs. Volcano Comparative Experiment
- Volcano Has Many Features, Why Is It Still Hard to Tune in Enterprises?
- Kubernetes or Slurm
- Running Kubernetes on KubeVirt: Architecture Boundaries, Production Design, and AI Scenario Trade-offs
- KubeVirt Single-Node Desktop Practice: Local Disks, CDI, and Browser noVNC
- Building a Persistent GPU Notebook with KubeVirt and Ceph RBD
- Spot GPU and Checkpoint Recovery Experiment
- Kubernetes Practice with Domestic GPU/NPU
End-to-End Delivery¶
- Large Model SFT Introduction: Explaining Loss, LoRA, Batch, and Parallelism in One Go
- Large Model SFT Training Practice: From Single-GPU LoRA to DeepSeek V4
- How Much Faster Can RDMA Make Distributed Training: DeepSeek V4 Dual-Node 16-GPU Test
- SwanLab Self-Hosting: From Kubernetes Deployment to Real SFT Metrics
- From W&B Local to SwanLab: Two-Year Team Experiment Tracking Practice and Selection
- Deploying MiniMax-Music3 with SGLang-Omni: From a Creative Idea to a Complete Song
- Qwen3.8-27B Day 0: vLLM and SGLang Test Records
- From Docker to Kubernetes: DeepSeek Harness, Built-in Chromium, and DSH Plugin Practice
- GLM-5.2 FP8 on 8×H20 141GB: AIBrix + vLLM Test
- DeepSeek-V4-Flash-0731 H20 Deployment and Stress Test
- Implementing AIBrix on an Existing Kubernetes Cluster
- AIBrix for Production: DS V4 Flash Stress Test, Dashboard, and Recovery Practice
- AIBrix Production Practice: Capacity, Exit, and Overload Protection for Qwen3.8-27B on L20
- What Does AIBrix Routing Optimize: From Cache Hits to Tail Latency
- AIBrix Real GPU Test: From Two-Node Inference to Eight-Node Fragmented GPUs
- Deploying ComfyUI on Kubernetes: Offline Images, CephFS Models, and Cross-Cluster Ingress
- RBG Multi-Role Inference Orchestration and sr1 Practice
- RBG with Traffic Practice: When Replicas Recover, Do Requests Recover?
- SGLang Model Gateway CPU Practice: Dynamic Discovery, Routing, and Fault Removal
- From Ollama to Kubernetes Production Inference
- Distributing a 70B Model to a Hundred Nodes
- Deploying AI/LLM Platform in Offline Environment
- Agent Sandbox Attack and Defense Experiment
Reading and Reproduction Conventions¶
Each experiment records at least the following:
- Kubernetes, driver, CUDA/ROCm/CANN, and component versions;
- GPU/NPU/TPU, CPU, memory, disk, and network topology;
- Model, precision, input/output length, and concurrency distribution;
- Cold cache/hot cache, failure retries, and warm-up status;
- Raw logs, events, metrics, traces, and timestamps;
- Success criteria, failure criteria, and the scope of inapplicability of results.
When reproducing experiments, first modify Namespace, images, StorageClass, resource names, and security policies. Do not directly copy permissions, image tags, or cloud provider parameters from the examples into production.