Koordinator and Online-Offline Workload Colocation: From Resource Reclamation to Production Deployment¶

Online services request resources based on peak usage, while offline tasks can tolerate queuing, throttling, and retries. Placing both types of workloads into the same set of machines can convert resources that online services do not actually use into batch processing throughput, but it also brings CPU, memory, cache, disk, and network contention into the same failure domain.

Therefore, colocation is not "squeezing in more Pods," nor is it simply raising Kubernetes resource overcommitment ratios. A usable colocation system must answer at least three questions simultaneously:

1. How much resource can still be borrowed? The resource ledger must distinguish between requested amounts, actual usage, safety margins, and capacity already lent out.
2. Where should tasks be placed? The scheduler must understand not only resource requests but also current node load, NUMA (Non-Uniform Memory Access) topology, and quotas.
3. What if the prediction is wrong? The node side must be able to throttle or evict lower-priority tasks within seconds to return resources to online services.

The value of Koordinator lies precisely in connecting these three questions into a closed control loop. On top of Kubernetes, it adds resource profiling, load-aware scheduling, differentiated QoS (Quality of Service), fine-grained CPU orchestration, elastic quotas, and safe rescheduling capabilities. This article is based on the community stable documentation v1.8; the in-development next documentation is used only to observe direction and is not a basis for production configuration.

1. Why Clusters Are "Fully Allocated" While Machines Are Not Running at Full Capacity¶

The Kubernetes scheduler primarily performs admission and placement based on a Pod's requests. To withstand traffic peaks, release jitter, and single-machine failures, online services typically configure request values higher than actual usage during most time periods. This practice guarantees capacity, but it also produces three distinct resource curves:

• Limit / Request: the upper bound declared by the business, or the scheduling reservation;
• Usage: current real consumption, varying with traffic;
• Reservation: the predicted reserved amount derived from historical usage, recent trends, and safety factors.

Request - Usage cannot all be lent out. Starting Pods, traffic bursts, metric latency, system processes, and prediction errors all require safety margins. A more reasonable approximation of reclaimable resources is:

Reclaimable resources = node allocatable resources
           - online workload prediction reservation
           - system and failure margin
           - resources already lent to low-priority workloads

可回收资源 = 节点可分配资源
           - 在线业务预测保留
           - 系统与故障余量
           - 已借给低优先级负载的资源

This value changes over time. CPU can be throttled, and recovery from incorrect predictions is usually fast; memory is incompressible, and overly aggressive predictions can trigger a full-machine OOM (Out of Memory). Disk I/O, network bandwidth, last-level cache, and memory bandwidth can also create hidden bottlenecks where "CPU does not look saturated, yet online latency degrades."

The Google Borg paper provides a highly representative production signal: in its median cell, approximately 20% of workloads run on reclaimed resources; when predictions are inaccurate, the system throttles or terminates non-production tasks rather than sacrificing production tasks. This shows that the premise of reclaimed capacity is not that predictions are always correct, but that errors must be quickly reversible, and borrowers must be able to yield.

2. The Object of Colocation Is Not the Two Labels "Online" and "Offline"¶

Production environments must decompose workloads into at least four dimensions:

Kubernetes native QoS classifies Pods into Guaranteed, Burstable, and BestEffort based on CPU and memory request/limit, primarily affecting eviction order under node pressure. It cannot fully express colocation semantics such as "online services can share CPU but need tail latency protection" or "batch tasks can use idle cores but must yield at any time."

Koordinator therefore separates Priority from QoS (Quality of Service) into two dimensions: priority determines queuing, preemption, and resource guarantee order; QoS determines how a Pod uses CPU, memory, cache, and bandwidth after landing on a node.

Koordinator QoS¶

Koordinator Priority¶

If a low-priority task lacks retry, checkpoint, or idempotency, merely labeling it with BE does not automatically make it suitable for colocation. Classification must come from business contracts, not unilateral guesses by the platform team.

3. How Koordinator Forms a Closed Control Loop¶

Koordinator's core components each handle problems at different time scales:

The closed loop sequence can be summarized as:

1. koordlet collects real usage of nodes and Pods, forming NodeMetric;
2. koord-manager computes batch-cpu, batch-memory, and other borrowable resources based on usage, safety thresholds, and configuration, then issues NodeSLO;
3. koord-scheduler selects nodes using the resource ledger, real-time load, and topology information;
4. koordlet and Runtime Hook implement CPU, memory, cache, I/O, and other isolation on the node;
5. When online load rises or node pressure exceeds bounds, throttle first, then evict low-priority workloads;
6. New usage and interference metrics feed into the next round of decisions.

This also explains why installing only koord-scheduler cannot be called complete colocation: the scheduler only decides the initial placement; load changes hours after running must be handled by node-side feedback control.

For how the node side implements policies into cgroups, resctrl, and the container runtime, continue reading Koordlet and Runtime Hooks: Node Resource Isolation.

4. Four Key Mechanisms¶

4.1 Dynamic Resource Ledger¶

Koordinator converts the portion of online Pod resources that have been requested but are predicted to go unused into batch resources. Batch tasks use a separate extended resource rather than continuing to consume the ordinary cpu and memory ledgers. This avoids double-counting the same capacity across two scheduling paths.

Resource profiling must be handled conservatively:

• Recent usage peaks and longer-window quantiles;
• Metric sampling latency and Pod startup period estimates;
• System overhead of kubelet, kernel, runtime, and DaemonSets;
• Capacity margins for single-machine failures, online scaling, and rolling releases;
• Different compressibility of CPU versus memory.

The official resource model distinguishes short-term from long-term Reservation: tasks with short lifecycles that are easy to retry can use more aggressive short-term margins; tasks with high recovery costs such as training and stream computing should use more stable long-term margins or explicit quotas.

4.2 Load-Aware and Topology-Aware Scheduling¶

The native scheduler sees request-allocated amounts; Koordinator's LoadAwareScheduling can additionally read NodeMetric and filter and score nodes by current or historical quantile load. When a new Pod has not yet produced metrics, the scheduler uses request, limit, and estimation coefficients to avoid misjudging a starting low-load Pod as idle.

For CPU-sensitive, GPU, RDMA (Remote Direct Memory Access), or high-throughput tasks, the following must also be handled:

• CPU physical cores and hyperthread sibling relationships;
• CPU, memory, and device alignment within NUMA;
• PCIe, GPU, NIC, and network topology;
• Resource fragmentation and schedulability of subsequent large-spec tasks.

Load balancing and Binpack are not contradictory: at the cluster level, low-load nodes can be packed as tightly as possible to free up entire machines, but hotspot filtering, safety thresholds, and topology constraints must be given higher priority.

4.3 Node-Side Isolation and the "Last Line of Defense"¶

Successful scheduling only means the workload could be placed at that moment. Production colocation relies more heavily on continuous node-side protection:

• CPU: CPUSet, CPU Burst, shared pool, dynamic throttling, Core Scheduling, or Group Identity;
• Memory: Memory QoS, asynchronous reclamation, priority-based reclamation, and threshold eviction;
• Cache/Bandwidth: Linux resctrl, LLC (Last Level Cache), and MBA (Memory Bandwidth Allocation);
• Disk and Network: rate limiting or guarantees by QoS class, preventing batch tasks from exhausting queues and bandwidth;
• Exception recovery: when throttling low-priority tasks still fails to relieve pressure, execute controlled eviction.

Memory protection in particular cannot simply copy CPU policy. When CPU is overestimated, it can be throttled; when memory is overestimated, synchronous reclamation and jitter may occur first, ultimately letting the kernel OOM Killer choose processes. The official documentation therefore recommends treating MemoryEvict as the last line of defense against node OOM in production colocation, and considering MemoryAllocatableEvict when using the batch resource ledger.

4.4 Quotas, Jobs, and Rescheduling¶

Colocation capacity is not "first come, first served until exhausted." Multi-team environments additionally require:

• ElasticQuota to express organizational hierarchy, minimum guarantees, maximum limits, and borrow-return relationships;
• Gang Scheduling to ensure distributed tasks are admitted as a group;
• Reservation to reserve resources for scaling, migration, scheduled jobs, or device tasks;
• Descheduler to migrate workloads after hotspots, topology changes, and long-term fragmentation;
• PodMigrationJob, PDB (Pod Disruption Budget), and migration rate limiting to jointly constrain disruption.

Without quota governance, low-priority tasks can still form "starvation within batch tasks" even when resources are abundant; without job semantics, distributed tasks may start only a subset of Workers, occupying resources while being unable to make progress.

5. What Large-Scale Enterprise Practice Tells Us¶

Public case studies differ in environment, hardware, business, and metric definitions; one company's utilization rate cannot be directly taken as one's own commitment value. What is more valuable is extracting common engineering constraints.

The boundaries of these solutions should also be noted: Gödel is a scheduler built by ByteDance for unified scheduling and ultra-large-scale Job semantics, not production data for Koordinator; the Koordinator Adopters list is a community self-reported inventory, not an independent audit. When citing these materials, the type of evidence should be stated, avoiding writing "has been used" as "has been proven suitable for all environments."

Common Pattern One: Define Who Can Be Sacrificed First¶

Borg, Alibaba's dual schedulers, and Koordinator's BE class all rest on the same premise: batch tasks accept longer startup times, resource fluctuation, or retries. The platform must implement this in job templates, retry budgets, checkpoint cycles, and business SLOs, rather than leaving it at priority numbers.

Common Pattern Two: Resource Ledgers Must Be Unified; Execution Can Be Layered¶

Kubernetes, YARN, or standalone batch schedulers can each retain their own submission entry points, but available capacity, allocated amounts, and return actions must converge on a single source of truth. The focus of YARN Copilot is not "putting NodeManager into a container," but having YARN use the batch resources computed by Koordinator, with both sides knowing who has occupied which resources.

Common Pattern Three: Utilization Is Not the Only Outcome¶

Real benefits must be observed simultaneously across:

• Online business P99/P999 latency, error rate, and throughput;
• Batch task completion time, queueing time, retry and eviction rate;
• Node CPU, memory, PSI (Pressure Stall Information), LLC, disk, and network;
• Predicted values, actual usage values, and fulfillment rate of reclaimable resources;
• Saved machine costs, additional platform costs, and failure costs.

A chart showing "average CPU rising from 20% to 55%" cannot prove that colocation succeeded. If online P99 degrades, batch tasks rerun every day, or the operations team frequently intervenes manually, the higher the book utilization, the greater the actual cost may be.

6. A More Reliable Production Rollout Path¶

Stage 0: Establish a Baseline Without Colocation¶

First, observe continuously for at least one complete business cycle and record:

• Online service request, usage, P50/P90/P99/P999, and peak/valley patterns;
• Node CPU, memory, PSI, disk, network, OOM, and Eviction;
• Batch task queue, runtime, failure rate, and retry cost;
• Special windows such as releases, scaling, failover, and major sales events.

If there is no baseline at all, after colocation you cannot determine whether latency fluctuations come from the new policy or from the business itself.

Stage 1: Online Protection Only, No Resource Reclamation¶

First validate QoS classification, CPU/NUMA orchestration, Memory QoS, metrics, and alerts. Select a small number of nodes and confirm that Koordinator's Runtime Hook and cgroup are compatible with the existing kubelet policy. This stage can surface differences in kernel, cgroup v1/v2, CPU Manager, container runtime, and operating system.

Stage 2: Let the Easiest-to-Recover Batch Tasks Enter the Pilot¶

Start with stateless, idempotent, short-running CPU batch tasks:

• Restrict to a dedicated node pool and a business whitelist;
• Set a maximum batch resource ratio for the pilot;
• Open CPU reclamation only, keep memory conservative;
• Clearly define suppression, eviction, retry, and exit conditions;
• Automatically stop new batch tasks when the online SLO is breached, and reclaim borrowed capacity.

Stage 3: Establish Resource Fulfillment Rate and Interference Matrix¶

Validate CPU-intensive, memory-intensive, disk-intensive, and network-intensive tasks category by category. Combine online services and batch tasks into an interference matrix, and record P99, PSI, cache hit rate, memory bandwidth, and task completion time before and after co-location on the same machine. Only combinations that have been validated may be expanded in scope.

Stage 4: Extend to Stateful Computing and Multiple Schedulers¶

Training, Spark, Flink, YARN, and similar tasks require additional Checkpoint, Gang, elastic quota, and unified accounting. Only at this point should you introduce YARN Copilot, Reservation, or more complex rescheduling policies, to avoid changing too many control loops at once in the first stage.

Stage 5: Turn Colocation into Continuous Capacity Engineering¶

Recalibrate thresholds according to business cycles; incorporate node hardware, kernel versions, and workload profiles into grouping; perform canary, rollback, and benefit review for every policy change. Colocation is not a one-time switch but a long-running feedback control system.

7. Which Metrics Should Production Acceptance Look At¶

Online Business Red Lines¶

Batch Task Experience¶

Platform Control Plane¶

A simple but practical benefit formula is:

Net benefit = saved machine and energy costs
       - online performance regression cost
       - batch task retry and checkpoint cost
       - colocation platform and operation cost

净收益 = 节省的机器与能耗成本
       - 在线性能回退成本
       - 批任务重试与 Checkpoint 成本
       - 混部平台和运维成本

8. Common Failure Modes¶

Only Looking at Average Utilization¶

Averages mask instantaneous hotspots and tail latency. Colocation thresholds should at minimum combine high-percentile usage, PSI, and online business SLOs.

Using the Same Oversell Ratio for CPU and Memory¶

CPU can be throttled, but memory pressure can cause synchronous reclamation and OOM. Memory policy should be more conservative, with a verifiable eviction path prepared.

Treating Running as Effective Throughput¶

A BE Pod may be in Running state but throttled to very little CPU. You should also look at task progress, resource satisfaction, and completion time.

Enabling Across the Entire Cluster at Once¶

Node hardware, kernel, cgroup, and workload profiles differ. Use node labels and layered SLO configuration for canary rollout, and keep a non-colocated control group.

Two Schedulers Each Calculating Idle Resources Separately¶

If Kubernetes and YARN, or online and batch schedulers, do not share a unified ledger, the same surplus can be allocated twice. Synchronization must cover total amount, allocated amount, Pod/Container exit, and abnormal recovery.

Only Limiting CPU Without Observing Cache, Memory Bandwidth, and I/O¶

Online latency can degrade even when CPU utilization is not high. Passing a CPU-intensive pilot does not mean disk- or network-intensive tasks are safe either.

Letting Non-Recoverable Tasks Bear Concession Semantics¶

A long task without idempotency, retry, or Checkpoint can lose hours of computation from a single eviction. It is better suited for guaranteed mid capacity than opportunistic free capacity.

9. When Is Koordinator Suitable¶

Koordinator is not a simple replacement for Kueue, Volcano, or Cluster Autoscaler: Kueue handles workload admission and quotas, Volcano excels at batch scheduling and Gang, and Cluster Autoscaler/Karpenter handles adding or reclaiming nodes. Koordinator focuses more on fine-grained scheduling, overselling, QoS, and interference governance over the same set of resources, and in production the two can be combined by boundary.

10. Minimal Viable Pilot Checklist¶

• Online services already have clear P99/P999, error rate, and capacity baselines;
• Batch tasks have idempotency, retry, or Checkpoint;
• Priority and QoS are confirmed by business owners;
• Pilot node pool, maximum borrowing ratio, and rollback method are clearly defined;
• A conservative policy is used when NodeMetric expires;
• CPU Suppress, Memory Evict, and node OOM paths have been through failure drills;
• Prometheus collects key metrics from both the control plane and pilot nodes;
• Online SLO breaches can automatically stop admission and release batch capacity;
• At least one group of control nodes without colocation enabled is retained;
• Net benefit is recalculated before every scope expansion, not just utilization.

11. Conclusion¶

The essence of colocation is turning idle capacity into a resource that can be borrowed, returned, and audited. Koordinator provides complete building blocks from resource profiling and scheduling to ledger, QoS, and eviction, but the tool itself will not define business priorities for the team, nor will it add recovery capability to batch tasks.

Large-scale practice has repeatedly validated the same thing: a successful colocation system does not require predictions to never be wrong, but requires hard protection for online services, concession capability for low-priority tasks, a closed resource ledger, and observability and rollback for every policy change. Get these four things solid first, then talk about utilization improvement — only then does colocation turn from a one-time cost experiment into a long-running capacity capability.

References¶

• Koordinator: Architecture Overview
• Koordinator: Resource Model
• Koordinator: QoS
• Koordinator: Batch Colocation Quick Start
• Koordinator: Load-Aware Scheduling
• Koordinator: Memory Resource Eviction
• Koordinator: Fine-Grained CPU Orchestration Practice
• Koordinator YARN Copilot Design
• Koordinator Adopters
• Kubernetes: Pod QoS Classes
• Google: Large-scale cluster management at Google with Borg
• Alibaba Cluster Trace Program
• CNCF: Gödel Scheduler open-sourced