AKS Pod Scheduling Demystified
1. The scheduling question Picture four Azure Kubernetes Service (AKS) worker nodes with the same hardware. One keeps receiving new batch Jobs even as its CPU usage climbs. The natural question is: why does Kubernetes send another pod to a node that already looks busy? The answer is that the scheduler does not work like a live CPU load balancer. Its normal resource checks rely on the requests declared by pods. The process has two decisions: Filtering: Which nodes can run the incoming pod? Scoring: Which of those eligible nodes is the best match according to scheduling rules? Binding and runtime come next. The example below follows the same pod through all five stages. 2. Capacity, allocatable, requests, and usage Term What it means Capacity The CPU and memory a node has. Allocatable The portion Kubernetes makes available to pods, after reservations for system services. Requests The CPU and memory a pod asks Kubernetes to reserve for scheduling. Actual usage The CPU and memory containers are using right now. Suppose a node has 16 allocatable CPU cores. Its running pods request 11 cores but are currently consuming 14 cores. Kubernetes counts 5 unrequested cores when deciding whether another pod fits, not 2 idle cores. A CPU limit is different from a request. A request influences placement; a limit can restrict how much CPU a container uses. 3. One pod from arrival to completion We'll follow one batch pod, batch-worker-01, across workers A, B, C, and D. Each worker has 16 allocatable CPU cores and 64 GiB allocatable memory. Figure 1. The entire scheduling journey, with decisions at filtering, scoring, runtime, and Job completion. Follow the same pod from top to bottom. Stage 1: Filtering. Can each node run the pod? A new pod arrives and asks for 0.25 CPU cores (250m) and 0.5 GiB memory (512Mi). The scheduler checks each worker. For CPU and memory, the fit rule uses the formular: Existing requests + Incoming pod request <= Node allocatable The rule is checked separately for CPU and memory. Required placement rules, including taints and affinity, must also pass. Worker Unrequested CPU before pod Unrequested memory before pod Can the 0.25 CPU / 0.5 GiB pod fit? A 4.80 cores 29 GiB Yes B 4.20 cores 32 GiB Yes C 4.40 cores 43 GiB Yes D 4.60 cores 34 GiB Yes Result: All four workers pass filtering. None has won yet. Their actual CPU usage has not entered this calculation. Stage 2: Scoring. Which eligible node ranks highest? For this walkthrough, look at the default NodeResourcesFit LeastAllocated resource-scoring calculation. It favors a node with more capacity not already requested by pods. We use equal CPU and memory weights of 1 each and MaxNodeScore = 100. The Kubernetes source implements the weighted calculation as: LeastAllocated score: Here A is node allocatable capacity, R is requested capacity including the incoming pod, and w is the resource weight. See the official Kubernetes LeastAllocated implementation and scheduler configuration reference. The denominator is 2 here because CPU and memory each have weight 1. The scheduler considers the incoming pod's request, so subtract 0.25 CPU and 0.5 GiB before scoring each candidate: Worker CPU left after pod Memory left after pod Formula using 16 CPU and 64 GiB allocatable Score / 100 A 4.55 28.5 GiB [(4.55/16 × 100) + (28.5/64 × 100)] / 2 36.5 B 3.95 31.5 GiB [(3.95/16 × 100) + (31.5/64 × 100)] / 2 37.0 C 4.15 42.5 GiB [(4.15/16 × 100) + (42.5/64 × 100)] / 2 46.2 D 4.35 33.5 GiB [(4.35/16 × 100) + (33.5/64 × 100)] / 2 39.8 For worker C: Worker C score after accounting for the incoming pod: Result: Worker C ranks first for this resource-scoring calculation. Worker A has slightly more unrequested CPU, but C's larger memory advantage wins under equal weighting. The default Kubernetes scheduler also has other scoring components, including resource balance and image locality. Thus, this is the resource-plugin score, not a guarantee of the final combined result. Stage 3: Binding. Where does the pod go? Assume the other scoring components do not change the ranking. The scheduler assigns batch-worker-01 to worker C. Result: The pod has a selected node. The kubelet on that node can now start its container. Stage 4: Runtime. Why can worker C still become busy? The pod's CPU request stays at 0.25 cores, even if its program starts consuming substantially more CPU. Suppose its workload uses 1.2 cores while it runs. Scheduler accounts for Pod actually consumes 0.25 CPU requested 1.2 CPU cores in use Worker C still has request-based headroom Worker C may be busy running other pods Future pods are checked against requests More overlapping work can increase CPU contention Try the CPU behavior in a test cluster. The following Pod runs two CPU-working Python processes for around 45 seconds. Unlike a sleep command, the processes actively use CPU. Their exact consumption depends on what CPU is available, but their request remains 250m. apiVersion: v1 kind: Pod metadata: name: batch-worker-01 labels: app: batch-cpu-demo spec: restartPolicy: Never containers: - name: worker image: python:3.12-alpine command: - python - -c - | import multiprocessing as mp import time def work(): end = time.monotonic() + 45 while time.monotonic() < end: pass processes = [mp.Process(target=work) for _ in range(2)] for p in processes: p.start() for p in processes: p.join() resources: requests: cpu: 250m memory: 512Mi Save as batch-worker.yaml, then run in an approved test namespace: kubectl apply -n <test-namespace> -f .\batch-worker.yaml kubectl get pod batch-worker-01 -n <test-namespace> -o wide kubectl top pod batch-worker-01 -n <test-namespace> --containers kubectl delete pod batch-worker-01 -n <test-namespace> Run kubectl top while the pod is active, and use Grafana if you want a graph of CPU consumption over time. This tests the difference between requested and actual CPU; a single pod will not demonstrate a lasting node-wide scheduling imbalance. Stage 5: Completion. What happens to its reservation? When the workload finishes and the completed pod stops counting toward active scheduling requests, its 0.25-core reservation is released. Its CPU usage also stops. Worker C may therefore look attractive again when the next Job arrives. A new Job begins a new filtering and scoring cycle. Worker C does not automatically win again, but repeated short-lived Jobs can keep a node busy while reservations are repeatedly freed and reused. The journey in one sentence: A pod fits on four nodes, worker C scores highest under the example resource score, the scheduler binds it there, it runs using more CPU than requested, then finishes and frees its reservation for the next scheduling cycle. 4. Why memory can outweigh CPU All four workers have the same 64 GiB allocatable memory, but their existing pods reserve different amounts: Worker Memory reserved by existing pods Unrequested memory A 35 GiB 29 GiB B 32 GiB 32 GiB C 21 GiB 43 GiB D 30 GiB 34 GiB Worker C does not have more physical memory. It simply has less memory requested by the pods already there. Under equal-weight LeastAllocated scoring, that can be enough to give C the higher resource score even if another worker has more unrequested CPU. 5. How to investigate an uneven placement pattern Start with these three questions: Which nodes were eligible? Compare allocatable resources, pod requests, taints, affinities, and pod-count limits. Why might one node score better? Compare CPU and memory requests, not just live usage. What happened after binding? Compare actual CPU usage, new pods per node, Job completions, and pod startup time. For current node state, use PowerShell and kubectl: kubectl config current-context kubectl get nodes -o wide kubectl describe node <node-name> kubectl describe node shows current state, not what the node looked like last week. In Grafana, look for historical kube_node_status_allocatable, kube_pod_container_resource_requests, kube_pod_info, and container_cpu_usage_seconds_total. Compare nodes at the same timestamps. Where available, use kube_pod_status_scheduled_time for pod scheduling times, rather than treating pod creation time as binding time. For control-plane evidence, AKS can export the kube-scheduler log category through diagnostic settings. These logs establish which pod was bound, where, and when, but do not always contain scoring details. See AKS monitoring resource logs. 6. How to prevent pods piling onto one node A few workload-level changes usually deserve attention before changing scheduler profiles: Right-size CPU requests. If pods commonly consume much more CPU than they request, scheduling can pack too much real work onto a node. Size requests from actual application measurements. Spread related pods. A hostname-based topologySpreadConstraints rule can encourage more even placement. ScheduleAnyway prefers spreading, while DoNotSchedule can block placements that exceed a configured skew. Control batch concurrency. Limit .spec.parallelism for individual Jobs, or use controller-side limits or Kueue admission for many independent Jobs. Reintroduce nodes carefully. Uncordoning immediately makes a node eligible for ordinary scheduling. Test with approved workloads and monitor placements before opening it fully. For example, a Job template can include: topologySpreadConstraints: - maxSkew: 2 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: batch-worker Remember that ScheduleAnyway is a preference, not a guarantee of equal pod counts. Test rules against your own workload mix. Changing to a bin-packing strategy such as MostAllocated is not an automatic fix for CPU-heavy workloads that under-request CPU. 7. Conclusion and references The Kubernetes scheduler starts with pod requests, not the node's live CPU graph. Filtering determines where the pod can run, and scoring decides which eligible node is preferred. In our example, worker C wins the resource score because its unrequested memory offsets a small CPU difference. Once the pod runs, its real CPU use may exceed what the scheduler reserved. Practical takeaway: Use realistic requests, spread workloads where balance matters, manage Job concurrency, and compare scheduler evidence with runtime metrics before deciding that a busy node is being scheduled incorrectly. Learn more Kubernetes scheduling framework Kubernetes scheduler configuration Kubernetes LeastAllocated scoring source Kubernetes resource management Kubernetes topology spread constraints AKS scheduler guidance AKS control-plane logs
