Deployments Reconcile Desired State

Kubernetes controllers continuously compare observed cluster state with a declared desired state. A Deployment is the most common expression of that: declare a pod template and a replica count, and a controller works until reality matches. Understanding which controller does what — and what each signal actually means — is most of understanding Kubernetes operations.

One image change, step by step

you: kubectl set image deploy/api api=registry/api@sha256:4f3a…c91e
   │
   ▼ Deployment controller
     compares spec.template hash → no matching ReplicaSet → creates one
     scales old ReplicaSet down, new one up, honoring the rollout strategy
   │  (default: maxSurge 25%, maxUnavailable 25%)
   ▼ scheduler          binds each Pending pod to a node
   ▼ kubelet + runtime  pulls image (5–30 s cold), starts container
   ▼ readiness probe    passes → pod added to Endpoints → receives traffic
   ▼ old pods           terminated after the new ones are ready (maxUnavailable: 0)

Surge and unavailability are percentages of the desired count, so with 6 replicas and maxUnavailable: 0, the rollout never dips below 6 serving pods — it briefly runs 7 (6 × 1.25 = 7.5 → 7). A pod-level rollout timing estimate for capacity planning:

per pod: image pull 8 s + start 2 s + readiness 3 × 5 s period = 25 s
6 replicas, one at a time (maxUnavailable 0) → ~2.5 min full rollout
adding 2 workers instead (parallel surge)    → ~1.5 min

That arithmetic tells you whether your change lands in 90 seconds or 15 minutes, and it tells your on-call how long "the deploy is stuck" actually means.

Probes are contracts, not pings

Three probes, three different meanings — confusing them is the most common self-inflicted outage on the platform:

readinessProbe:                  # "am I ready to serve THIS request?"
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3            # out of traffic after ~15 s of failures
livenessProbe:                   # "is this process worth restarting?"
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3            # restart after ~30 s of failures
startupProbe:                    # "how long may I boot before being called dead?"
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 10              # up to 5 min of slow startup

The failure modes are symmetrical. A liveness probe that checks a dependency (database reachable) restarts healthy pods during a database blip — turning a partial failure into PodsInCrashLoopBackOff across the fleet. A readiness probe that returns false forever keeps a running, correct process out of the load balancer while dashboards show everything green. The contract: liveness = this process can never make progress; readiness = don't send me requests right now. Startup probes exist so a 90-second JVM boot is not killed by a liveness probe with a 30-second patience.

Six desired, four ready: what to inspect

This is the check question, and the signals split cleanly across three layers:

cluster side (kubectl describe pod / events)
  - Pending + "Insufficient cpu/memory"     → scheduling: node capacity or requests too big
  - Pending + "0/7 nodes available"         → cluster-wide pressure / taints
  - ContainerCreating + ImagePullBackOff    → tag/digest/registry auth
  - CrashLoopBackOff                        → probe fails or process exits (see logs)
  - terminated, OOMKilled (137)             → memory limit, not replica count
  - terminating, stuck                      → PDB blocking eviction, or slow preStop
 
application side
  - readiness failing: which endpoint, what error, how long?  ← pods run but don't serve
  - readiness passing but requests failing: Endpoints/proxy lag or wrong port
 
capacity side
  - all 6 scheduled and ready → you don't need more replicas, you need
    a diagnosis; scaling hides the fault and doubles its blast radius

The rule the table encodes: increasing replicas is a capacity decision, not a remediation step. If two of six are unhealthy, six more unhealthy pods are the likely outcome of the scale-up. The same applies to restarting pods on a liveness failure you have not yet diagnosed: a restart loop hides the crash reason behind a fresh container each time. Read the events and logs first; the controller is already doing its job of replacing what fails — your job is to stop it from failing.

Costs and failure cases

• A manifest is a request, not proof. apply returning success means the object was stored; it says nothing about image pulls, scheduling, readiness, or traffic.
• Rollouts stall for three usual reasons: image pull failures, insufficient allocatable capacity (requests vs actual node size), and readiness that never passes because the probe tests something the new version can't satisfy (config, migration, dependency).
• Aggressive probes restart healthy slow-starting processes in a loop; permissive liveness lets a wedged process hold a slot forever. Both show up as "the pod is running but the service is down."

Key takeaways

• Desired vs ready vs serving are three different numbers. rollout status, Endpoints, and your own request metrics answer each in turn.
• Surge math sets your rollout duration — per-pod time × replicas ÷ parallelism. Compute it once per service and stop being surprised by 15-minute deploys.
• Liveness means "restart me," readiness means "unroute me," startup means "wait for me." Point liveness at process health only, and never at a dependency.
• Six requested / four ready is a diagnosis, not a capacity problem. Read the events before you scale.

Kubernetes Documentation: Deployments defines the rollout strategy and probe semantics precisely.