A deploy replaces every instance of your service, one by one. If killing an instance also kills its 40 in-flight requests, users see errors on every deploy — that is what graceful shutdown exists to prevent. This trace follows one worker from the moment the orchestrator sends SIGTERM to the moment it exits with status 0.
1. SIGTERM is delivered
Kubernetes (or systemd, or the container runtime) wants this instance gone: kill -TERM <pid> is delivered by the kernel to the process. The default action for SIGTERM is immediate death — which is exactly what most services accidentally get. A service that never installs a handler is killed mid-request on every deploy: the TCP connections drop, clients get connection resets, and retries (see the retry trace) have to paper over a problem that has a trivial fix. The first step is registering a handler: signal.NotifyContext(ctx, syscall.SIGTERM) / addSignalHandler / Runtime.getRuntime().addShutdownHook.
2. The handler flips the state to draining
The handler sets a process-wide flag: state = DRAINING. From this instant the service is in a two-world mode: it must stop starting work while finishing what it started. The distinction is the whole design. A shutdownContext with a deadline (the orchestrator's grace period, typically 30-60s) starts ticking.
3. Readiness flips to draining
The instance tells the world it is leaving: /healthz still returns 200 (it's alive), but /readyz — the readiness probe that the load balancer and service mesh poll every few seconds — now returns 503. Within one poll interval (typically 5-10s with a 1-2s interval), the LB stops routing new connections here. Existing connections keep flowing; new work stops. This is why the LB needs to see "draining" before the listener closes — the order matters and gets it wrong in almost every homegrown implementation.
4. The listener stops accepting
The server calls httpServer.Shutdown(ctx) (Go), app.stop() (Node), or closes the socket. The kernel socket is closed, the accept queue is no longer fed — new TCP connections are refused at the kernel level (RST). Anything that arrived in the window between readiness-flip and socket-close is the normal small leak of a deploy. From this point, the process contains only: in-flight requests, background workers, and scheduled jobs.
5. In-flight requests complete under a deadline
The server now gives each in-flight request a chance to finish — this is the grace period's real work. A request mid-query (a 300ms DB call), mid-response (a 10MB download), or mid-webhook (see the webhook trace) completes normally. The contract: requests must finish within the grace period (30-60s); ones that don't are cut at the deadline. The server framework's Shutdown() blocks until either all requests finish or the context deadline fires. Requests still processing at the deadline get their connections closed — half-answered, exactly as if the deploy never happened, but bounded.
6. Background work drains
With the listener down, the process still owns: the connection pool (possibly with checked-out connections and queued waiters — see the connection pool trace), in-flight cron jobs and batch runs (see the cron trace), and worker loops consuming queues. Draining each is a policy: the pool close() waits for checked-out connections (bounded by its own timeout); the queue consumer stops fetching but finishes its current batch; the cron scheduler cancels its timer and lets an overlapping run finish or marks it interrupted. The unspoken rule: anything that can be checkpointed (batch jobs) should be told to checkpoint and exit; anything that can't should record "interrupted" state so a supervisor can resume it.
7. Exit with status 0
The process finishes: all goroutines/threads joined, the signal handler has completed, os.Exit(0). The orchestrator sees a clean exit — no restart backoff, no crash-loop, no "last deploy caused a cascade" postmortem. If the process is still alive when the grace period expires, the orchestrator sends SIGKILL — a hard cut — because a hung process must not block a rollout forever. A service whose shutdown respects the deadline exits first, every time.
The cost summary
Graceful shutdown is free at steady state: a few ms of flag-setting and one readiness blip per instance per deploy. Its entire value is the bounded tail — the difference between "40 requests lost per deploy" and "0 requests lost, 5s extra per rollout." The trace reduces to three decisions: stop accepting, finish in flight, exit on time — and the last one is the one most services get wrong.