https://aik8s.run/ GPU、大数据、调度、训练、推理、RAG、Agent 与生产运维 zh-CN Thu, 27 Aug 2026 04:16:37 +0000 https://aik8s.run/ai-k8s/practices/qwen38-flash-next-sglang-day0/ https://aik8s.run/ai-k8s/practices/qwen38-flash-next-sglang-day0/ Thu, 27 Aug 2026 04:15:59 +0000 用 SGLang Day-0 官方镜像和 BF16/FP8 权重，在 4×H20 上验证 Qwen4 预览架构、精度与容量、262K 长上下文、PLE、MTP、长短混部与框架支持边界 practices https://aik8s.run/ai-k8s/practices/ https://aik8s.run/ai-k8s/practices/ Thu, 27 Aug 2026 02:08:02 +0000 用可复现的实验、性能数字、事故证据和决策树回答 AI/LLM on Kubernetes 的高频问题 practices https://aik8s.run/ai-k8s/practices/gpu-topology-fragmentation-scheduling/ https://aik8s.run/ai-k8s/practices/gpu-topology-fragmentation-scheduling/ Wed, 26 Aug 2026 10:29:19 +0000 从 NVIDIA Device Plugin、Topology Manager、Volcano、Koordinator、HAMi 到 Kubernetes DRA，评价 GPU 设备拓扑感知调度的真实能力与生产落地路径 practices https://aik8s.run/ai-k8s/gpu-scheduling/ https://aik8s.run/ai-k8s/gpu-scheduling/ Wed, 26 Aug 2026 10:29:19 +0000 整卡、MIG、共享、拓扑、亲和性和资源碎片治理 ai-k8s https://aik8s.run/ai-k8s/training/wandb-vs-swanlab/ https://aik8s.run/ai-k8s/training/wandb-vs-swanlab/ Wed, 26 Aug 2026 03:31:00 +0000 从一套运行两年的 W&B Local 讲起，结合 SwanLab 真实 SFT 实测，解释实验追踪为什么难以生产化，以及团队应该怎样选型和验收 training https://aik8s.run/ai-k8s/training/swanlab-self-hosted/ https://aik8s.run/ai-k8s/training/swanlab-self-hosted/ Wed, 26 Aug 2026 03:23:50 +0000 在 Kubernetes 部署 SwanLab，接入 ms-swift，区分实验追踪与基础设施监控，并用 Qwen3.5、Qwen3.6 MoE 和 DeepSeek V4 的真实 SFT 验证指标链路与过拟合拐点 training https://aik8s.run/ai-k8s/ https://aik8s.run/ai-k8s/ Wed, 26 Aug 2026 03:23:50 +0000 从加速器、大数据和集群，到训练、推理、RAG、Agent 与生产运维的完整技术地图 ai-k8s https://aik8s.run/ https://aik8s.run/ Wed, 26 Aug 2026 03:23:50 +0000 面向平台工程师、SRE、数据平台和模型服务团队的 AI/LLM 基础设施工程文档 首页 https://aik8s.run/ai-k8s/training/sft-from-single-gpu-to-deepseek-v4/ https://aik8s.run/ai-k8s/training/sft-from-single-gpu-to-deepseek-v4/ Tue, 25 Aug 2026 01:30:09 +0000 用 Qwen3.5-4B 跑通 LoRA、SwanLab 与盲测闭环，再用 8 至 16 张 H20 实测 DeepSeek V4 Flash 的 Expert Parallel、跨机 GDRDMA 与过拟合拐点 training https://aik8s.run/ai-k8s/training/rdma-distributed-training-benchmark/ https://aik8s.run/ai-k8s/training/rdma-distributed-training-benchmark/ Tue, 25 Aug 2026 01:30:09 +0000 用 NCCL AllReduce/All-to-All 微基准、DeepSeek V4 Flash MoE SFT 正负对照和 SwanLab 六轮 Run，解释哪些并行方式能真正从 RDMA 获益 training https://aik8s.run/ai-k8s/training/sft-concepts/ https://aik8s.run/ai-k8s/training/sft-concepts/ Mon, 24 Aug 2026 16:27:03 +0000 面向第一次接触大模型训练的读者，用一个完整例子解释 SFT、Token、Loss、LoRA、QLoRA、Batch、Epoch、Checkpoint、Dense、MoE、DP、TP、PP、EP、ZeRO 和效果评测 training https://aik8s.run/ai-k8s/distributed-training/ https://aik8s.run/ai-k8s/distributed-training/ Mon, 24 Aug 2026 16:27:03 +0000 训练控制器、Gang Scheduling、集合通信、容错和作业生命周期 ai-k8s https://aik8s.run/ai-k8s/training/kubernetes-domain-sft-community/ https://aik8s.run/ai-k8s/training/kubernetes-domain-sft-community/ Mon, 24 Aug 2026 13:15:06 +0000 分析 KubeFix、K8s Distill、KubeBench、ITBench 等社区实践，解释 Kubernetes 文档问答、Manifest 生成和故障诊断应怎样准备数据、训练模型并完成可信评测 training https://aik8s.run/ai-k8s/practices/sglang-0518-release-analysis/ https://aik8s.run/ai-k8s/practices/sglang-0518-release-analysis/ Mon, 24 Aug 2026 08:32:39 +0000 使用官方镜像在同一张 L20 上实测 SGLang v0.5.16、v0.5.17 与 v0.5.18，并对照官方 Release 分析吞吐、尾延迟、启动与容量变化 practices https://aik8s.run/ai-k8s/observability/ https://aik8s.run/ai-k8s/observability/ Fri, 21 Aug 2026 04:25:16 +0000 从用户请求和训练作业到 Pod、GPU、网络与存储的指标和追踪体系 ai-k8s https://aik8s.run/ai-k8s/practices/qwen38-27b-day0/ https://aik8s.run/ai-k8s/practices/qwen38-27b-day0/ Wed, 19 Aug 2026 09:00:27 +0000 记录 Qwen3.8-27B 官方 FP8 制品、vLLM 与 SGLang 镜像、单卡 L20 Kubernetes 实测、OpenWebUI 接入，以及后续 MTP 和长上下文 A/B practices https://aik8s.run/ai-k8s/practices/glm52-fp8-h20-aibrix-vllm/ https://aik8s.run/ai-k8s/practices/glm52-fp8-h20-aibrix-vllm/ Tue, 18 Aug 2026 01:23:52 +0000 记录 GLM-5.2 FP8 单节点 TP=8 的 vLLM 与 SGLang 基线，以及双节点 P/D 分离下 TCP 和 RDMA 的 KV 传输与端到端性能对比 practices https://aik8s.run/ai-k8s/practices/deepseek-v4-flash-h20-evaluation/ https://aik8s.run/ai-k8s/practices/deepseek-v4-flash-h20-evaluation/ Tue, 18 Aug 2026 01:23:52 +0000 记录 DeepSeek-V4-Flash-0731 在单机八卡 H20 上的资源条件、vLLM 启动流程、性能基线、同 Pod PD 分离实测与 OpenWebUI 对接方法 practices https://aik8s.run/ai-k8s/practices/minimax-music3-sglang-omni/ https://aik8s.run/ai-k8s/practices/minimax-music3-sglang-omni/ Mon, 17 Aug 2026 07:32:12 +0000 在 Kubernetes 单卡部署 MiniMax-Music3，并接入 OpenAI 兼容大模型自动生成歌名、结构化歌词和编曲描述 practices https://aik8s.run/ai-k8s/practices/qwen38-27b-h20-day0/ https://aik8s.run/ai-k8s/practices/qwen38-27b-h20-day0/ Sat, 15 Aug 2026 06:31:00 +0000 该文章已迁移到新地址 practices https://aik8s.run/ai-k8s/practices/nccl-unhandled-cuda-error-host-memory-oom/ https://aik8s.run/ai-k8s/practices/nccl-unhandled-cuda-error-host-memory-oom/ Fri, 14 Aug 2026 10:51:58 +0000 一次 ncclCommInitRank unhandled cuda error 的脱敏排障实录：从 GPU、PCIe、RDMA 逐步追到 cgroup OOM 与 NVIDIA 锁页失败 practices https://aik8s.run/ai-k8s/practices/gpu-failure-atlas/ https://aik8s.run/ai-k8s/practices/gpu-failure-atlas/ Fri, 14 Aug 2026 10:51:58 +0000 用 XID、ECC、NVLink、掉卡、NCCL、RDMA 和 kubelet 证据定位 GPU 故障 practices https://aik8s.run/ai-k8s/rag-agent/deepseek-harness-runtime-containerization/ https://aik8s.run/ai-k8s/rag-agent/deepseek-harness-runtime-containerization/ Fri, 14 Aug 2026 08:48:52 +0000 为 DeepSeek Harness 构建多架构 Docker 镜像，并用 Docker Compose 与 Kubernetes StatefulSet 安全部署，解决 EROFS、持久化、原生模块和 Web 暴露问题 rag-agent https://aik8s.run/ai-k8s/practices/deepseek-harness-kubernetes/ https://aik8s.run/ai-k8s/practices/deepseek-harness-kubernetes/ Fri, 14 Aug 2026 08:48:52 +0000 把 DeepSeek Harness、浏览器桌面和自研 DSH Plugin 封装到同一 OCI 镜像，记录 Kubernetes StatefulSet、PVC、NetworkPolicy、Cilium 探针与 Tailscale 远程访问边界 practices https://aik8s.run/ai-k8s/rag-agent/deepseek-harness-repository-analysis/ https://aik8s.run/ai-k8s/rag-agent/deepseek-harness-repository-analysis/ Fri, 14 Aug 2026 00:00:00 +0000 从源码目录、Cordis 插件树、Profile 与 Bundle、Agent 轮次、事件溯源会话和能力 seam，系统拆解 DeepSeek Harness rag-agent https://aik8s.run/ai-k8s/practices/model-weight-delivery-acceleration/ https://aik8s.run/ai-k8s/practices/model-weight-delivery-acceleration/ Thu, 13 Aug 2026 14:24:28 +0000 从对象存储、并行文件系统、P2P、节点缓存到 GPU Loader，拆解大模型权重交付链路并给出可执行的选型与测试路线 practices https://aik8s.run/ai-k8s/rag-agent/workbuddy-enterprise-agent-stack/ https://aik8s.run/ai-k8s/rag-agent/workbuddy-enterprise-agent-stack/ Thu, 13 Aug 2026 13:36:45 +0000 从腾讯 WorkBuddy 的 Agent、Runtime、Session、Skill、MCP 与评测能力，拆解企业 Agent 从模型到沙箱、身份、观测和交付的完整技术栈 rag-agent https://aik8s.run/ai-k8s/practices/model-distribution-100-nodes/ https://aik8s.run/ai-k8s/practices/model-distribution-100-nodes/ Thu, 13 Aug 2026 06:51:08 +0000 为 TB 级模型和百节点推理池设计 Registry、对象存储、P2P 与节点缓存实验 practices https://aik8s.run/ai-k8s/practices/llm-cold-start-optimization/ https://aik8s.run/ai-k8s/practices/llm-cold-start-optimization/ Thu, 13 Aug 2026 06:51:08 +0000 从 DeepSeek-V4-Pro 滚动重启接近半小时的真实问题出发，按启动关键路径拆解怎样将 H20 节点缓存重启优化到五分钟量级，以及模型缓存、权重加载、JIT、CUDA Graph 与热恢复的工程取舍 practices https://aik8s.run/ai-k8s/practices/distributed-kv-cache-deepseek-v4/ https://aik8s.run/ai-k8s/practices/distributed-kv-cache-deepseek-v4/ Thu, 13 Aug 2026 03:04:54 +0000 区分 P/D KV 直传、KV Offload、Prefix 感知路由与跨引擎共享缓存，比较 Mooncake、AIBrix、LMCache/llm-d 和 Dynamo KVBM，并给出 DeepSeek V4 Flash 的渐进验证路线 practices https://aik8s.run/ai-k8s/practices/aibrix-existing-cluster/ https://aik8s.run/ai-k8s/practices/aibrix-existing-cluster/ Wed, 12 Aug 2026 09:58:54 +0000 在 Kubernetes 1.30 集群安装 AIBrix v0.7.0，用 CPU mock 跑通模型路由、P/D、StormService、自动扩缩容、Prometheus 指标和 Higress 两层网关串联 practices https://aik8s.run/assets/practices/llm-cold-start-optimization/README/ https://aik8s.run/assets/practices/llm-cold-start-optimization/README/ Wed, 12 Aug 2026 08:07:48 +0000 The figures visualize the cold-start stages, the measured local storage A/B, and the optimization ladder documented in docs/ai-k8s/practices/llm-cold-start-optimization.md. assets https://aik8s.run/ai-k8s/rag-agent/internal-agent-app-publishing/ https://aik8s.run/ai-k8s/rag-agent/internal-agent-app-publishing/ Wed, 12 Aug 2026 04:17:06 +0000 面向商务、法务和运营等非研发用户的 AI 应用搭建、试运行、审批、发布、权限与运行时设计 rag-agent https://aik8s.run/ai-k8s/practices/rbg-existing-cluster/ https://aik8s.run/ai-k8s/practices/rbg-existing-cluster/ Tue, 11 Aug 2026 22:22:16 +0000 在 Kubernetes 1.30 集群部署 RoleBasedGroup，实测角色依赖、服务发现、扩缩、自愈、Ray 两机推理和 NIXL P/D 分离，并以相同镜像与模型对比 RBG 和 AIBrix practices https://aik8s.run/ai-k8s/rag-agent/agent-landscape-2026/ https://aik8s.run/ai-k8s/rag-agent/agent-landscape-2026/ Tue, 11 Aug 2026 20:25:55 +0000 主流编程、浏览器与企业 Agent 的产品格局、实现原理、开发框架、风险和演进趋势 rag-agent https://aik8s.run/assets/practices/deepseek-v4-flash-h20-evaluation/README/ https://aik8s.run/assets/practices/deepseek-v4-flash-h20-evaluation/README/ Tue, 11 Aug 2026 09:49:25 +0000 The figures are generated from the benchmark numbers recorded in docs/ai-k8s/practices/deepseek-v4-flash-h20-evaluation.md. assets https://aik8s.run/assets/practices/gpu-notebook-platform-evolution/README/ https://aik8s.run/assets/practices/gpu-notebook-platform-evolution/README/ Tue, 11 Aug 2026 09:01:01 +0000 The three article figures are generated by scripts/generategpunotebookfigures.py. assets https://aik8s.run/ai-k8s/practices/gpu-notebook-platform-evolution/ https://aik8s.run/ai-k8s/practices/gpu-notebook-platform-evolution/ Tue, 11 Aug 2026 09:01:01 +0000 从 JupyterHub/KubeSpawner、code-server 和组级八卡共享，走到 Ceph RBD、CephFS、Workspace Operator 与 KubeVirt 根盘的真实演进 practices https://aik8s.run/ai-k8s/development/gpu-notebook-platform/ https://aik8s.run/ai-k8s/development/gpu-notebook-platform/ Tue, 11 Aug 2026 09:01:01 +0000 从 JupyterHub、Kubeflow 和托管 Workbench，到整卡、MIG、共享 GPU、用户 Home、对象存储与本地缓存的生产选型 development https://aik8s.run/ai-k8s/rag-agent/openclaw-enterprise-agent-platform/ https://aik8s.run/ai-k8s/rag-agent/openclaw-enterprise-agent-platform/ Tue, 11 Aug 2026 00:00:00 +0000 分析 OpenClaw 的可复用能力、企业平台缺口、安全风险、推荐架构和采用决策 rag-agent https://aik8s.run/ai-k8s/agentic-workloads/ https://aik8s.run/ai-k8s/agentic-workloads/ Mon, 10 Aug 2026 16:16:11 +0000 Agent 工作负载的运行时隔离、网络、身份、工具权限和审计设计 ai-k8s https://aik8s.run/ai-k8s/practices/aibrix-gpu-multinode-pd-production/ https://aik8s.run/ai-k8s/practices/aibrix-gpu-multinode-pd-production/ Sat, 08 Aug 2026 13:22:34 +0000 在生产 Kubernetes 集群使用 NVIDIA L20、AIBrix v0.7.0、RayClusterFleet、StormService 和 vLLM，验证两机模型并行、NIXL P/D 分离、八节点 Qwen3-235B FP8，以及 DeepSeek 70B 的四节点单卡、两节点双卡和单节点四卡拓扑 practices https://aik8s.run/ai-k8s/practices/comfyui-minimax-h3-gpu/ https://aik8s.run/ai-k8s/practices/comfyui-minimax-h3-gpu/ Sat, 08 Aug 2026 12:42:42 +0000 使用 NVIDIA GPU、只读 CephFS、ComfyUI extra_model_paths、Init Container 模型别名及双层 Ingress，在受限网络 Kubernetes 环境提供 MiniMax-H3 工作流页面 practices https://aik8s.run/ai-k8s/inference/higress-ai-gateway/ https://aik8s.run/ai-k8s/inference/higress-ai-gateway/ Fri, 07 Aug 2026 21:59:40 +0000 在既有 Kubernetes 集群隔离安装 Higress，理解 Controller、Gateway、Console、AI Proxy 与可观测插件，并设计 Higress 和 AIBrix 的同集群及跨集群链路 inference https://aik8s.run/ai-k8s/practices/sglang-model-gateway-cpu-lab/ https://aik8s.run/ai-k8s/practices/sglang-model-gateway-cpu-lab/ Fri, 07 Aug 2026 09:53:15 +0000 在没有 GPU 的 Kubernetes 1.30 集群部署 SGLang Router 和两个 OpenAI-Compatible Mock Worker，实测动态发现、轮询、摘除恢复与 Prometheus 指标 practices https://aik8s.run/ai-k8s/inference/engines/ https://aik8s.run/ai-k8s/inference/engines/ Fri, 07 Aug 2026 09:53:15 +0000 对比 vLLM、SGLang、TensorRT-LLM、Triton、llama.cpp 等引擎的能力、边界和 Kubernetes 集成方式 inference https://aik8s.run/ai-k8s/inference/distributed-serving/ https://aik8s.run/ai-k8s/inference/distributed-serving/ Fri, 07 Aug 2026 09:53:15 +0000 设计多机模型副本、LeaderWorkerSet、Prefill/Decode 分离和 KV 传输，并对比 AIBrix、llm-d、KServe、Dynamo、Ray Serve 与 vLLM Production Stack inference https://aik8s.run/ai-k8s/inference/gateway-routing/ https://aik8s.run/ai-k8s/inference/gateway-routing/ Thu, 06 Aug 2026 12:12:15 +0000 区分 API Gateway、Gateway API Inference Extension 和模型请求调度，设计认证、配额、缓存感知和发布策略 inference https://aik8s.run/ai-k8s/practices/gpu-resource-bank-tidal-platform/ https://aik8s.run/ai-k8s/practices/gpu-resource-bank-tidal-platform/ Wed, 05 Aug 2026 10:14:41 +0000 用 Kueue 或 Volcano 实现业务组 GPU 配额互借，结合 KServe、AIBrix、KEDA 与 vLLM 构建实时和定时弹性，并建立从硬件健康到单位 Token 成本的可观测闭环 practices https://aik8s.run/ai-k8s/practices/kubevirt-local-desktop-lab/ https://aik8s.run/ai-k8s/practices/kubevirt-local-desktop-lab/ Wed, 05 Aug 2026 07:07:09 +0000 在没有 Ceph 的 Kubernetes 集群中，把 KubeVirt VM 限制到单个节点，使用本地盘保存完整系统环境，并通过受限 noVNC 网关从浏览器访问桌面 practices