A strong answer
Look inside kube-scheduler and the Eviction API: (1) The Two-Phase Scheduling Pipeline (Single-Threaded Scheduling Cycle + Concurrent Binding Cycle): For each unscheduled Pod in the Priority Queue, kube-scheduler takes a lock-free snapshot of its node cache and runs the synchronous Scheduling Cycle: PreFilter $\to$ Filter (Predicates: NodeResourcesFit, NodeAffinity, TaintToleration, PodTopologySpread—which nodes can run this Pod?) $\to$ PostFilter (Preemption: if 0 nodes fit, can evicting lower-priority Pods on a node make room without violating a PDB if possible?) $\to$ Score (Priorities: each plugin scores feasible nodes 0..100, multiplied by weight; highest total score wins!) $\to$ Reserve (optimistically marks resources as used in the in-memory cache!). Then the asynchronous Binding Cycle runs concurrently in a goroutine to write the Binding object to kube-apiserver! On a 5,000-node cluster, percentageOfNodesToScore stops filtering once ~10%–50% of feasible nodes are found so scheduling stays < 5 ms! (2) topologySpreadConstraints (maxSkew: 1): Setting topologySpreadConstraints across both topologyKey: topology.kubernetes.io/zone and kubernetes.io/hostname guarantees kube-scheduler never stacks all replicas on 1 node or 1 AZ! (3) PodDisruptionBudget (PDB) & the Eviction API: kubectl drain does not call raw DELETE /pods; it calls the POST /pods/<name>/eviction subresource, which atomically checks the PDB (disruptionsAllowed = currentHealthy - minAvailable). If replicas == 3 and minAvailable == 3 (or maxUnavailable: 0), disruptionsAllowed is 3 - 3 = 0, permanently blocking every voluntary node drain!
For any PodDisruptionBudget protecting a replica set with currentHealthy pods and threshold minAvailable (or desired - maxUnavailable), the Kubernetes Eviction API permits a voluntary pod eviction if and only if currentHealthy - minAvailable >= 1 (status.disruptionsAllowed >= 1). Because kube-apiserver processes Eviction requests using an optimistic-concurrency update on the PodDisruptionBudget status (disruptionsAllowed) and rejects with HTTP 429 Too Many Requests whenever currentHealthy <= minAvailable, concurrent kubectl drain operations across multiple nodes can never voluntarily reduce healthy replicas below minAvailable.
Using percentageOfNodesToScore (scoring a bounded sample of feasible nodes instead of all 5,000 nodes) keeps kube-scheduler throughput above 150–300 Pods/sec (~3 ms per Pod) even on 5,000-node clusters. In production systems (Kubernetes kube-scheduler Scheduling Framework, Karpenter node bin-packing & consolidation, Descheduler, and the Kubernetes policy/v1 PodDisruptionBudget & Eviction API.), engineers monitor these exact trade-offs to keep p99 latency predictable and prevent resource exhaustion under peak load.
What the interviewer is listening for
- Traces how the
kube-schedulerFramework (Filter -> Score -> Preempt -> Reserve -> Bind), Topology Spread, andPodDisruptionBudgets(PDB) operates in memory step by step rather than giving a vague textbook definition. - States the core correctness invariant and explains why edge cases do not violate it.
- Quantifies time, space, and hardware costs (
Filter+Score(percentageOfNodesToScore),topologySpreadConstraints(maxSkew: 1)) with concrete numbers. - Identifies real-world failure modes (Using
requiredDuringSchedulingIgnoredDuringExecutionpodAntiAffinityonkubernetes.io/hostnamefor a 50-replica Deployment in a 20-node cluster) and explains how production systems mitigate them.
Common mistakes
- Treating the
kube-schedulerFramework (Filter -> Score -> Preempt -> Reserve -> Bind), Topology Spread, andPodDisruptionBudgets(PDB) as a black box without explaining the underlying memory or state transitions. - Ignoring worst-case tail latency or memory overhead (Creating a
PodDisruptionBudgetwithminAvailable: 1on a Deployment that hasreplicas: 1). - Applying the pattern without enforcing explicit capacity bounds or validation (Remembering that Kubernetes
Scheduleronly runs once at Pod creation time (IgnoredDuringExecution), so after nodes scale up and down over a week, Pods end up skewed (10on Node A,0on Node B)).