Trace what happens inside `kube-apiserver` and `etcd` when you run `kubectl apply -f deployment.yaml`. Explain `etcd`'s MVCC `ModRevision`.

A strong answer

Every Kubernetes object's metadata.resourceVersion is literally the 64-bit monotonic ModRevision integer from etcd's MVCC key-value store (BoltDB/bbolt B+ Tree keyed by (major_revision, sub_revision)): (1) Optimistic Concurrency Control (409 Conflict): Kubernetes never holds pessimistic database locks across HTTP calls! When Controller A PUTs Deployment/api with resourceVersion: "1042", kube-apiserver issues an atomic etcd Compare-And-Swap (Txn where ModRevision(key) == 1042), creating revision 1043. When Controller B's PUT arrives carrying resourceVersion: "1042", the CAS fails (1042 != 1043) and kube-apiserver rejects it with HTTP 409 Conflict (forcing Controller B to re-read revision 1043 or use Server-Side Apply!). (2) kube-apiserver Watch Cache (Ring Buffer) & Bookmarks: Instead of opening 10,000 watches directly on etcd, kube-apiserver maintains an in-memory Watch Cache ring buffer fed by 1 watch per resource type on etcd. Clients run LIST (getting snapshot at resourceVersion = R) followed by WATCH ?resourceVersion=R (streaming all mutations > R with zero missed events!). To prevent an idle watch's resourceVersion from falling behind etcd's periodic compaction horizon (which causes HTTP 410 Gone and a costly full LIST relist storm!), allowWatchBookmarks=true periodically sends lightweight BOOKMARK events advancing the client's resourceVersion to the latest revision even when no objects matched its filter!

A PUT (update) to kube-apiserver carrying metadata.resourceVersion = R succeeds if and only if the target key's current ModRevision in etcd is still exactly $R$, upon which etcd atomically increments the cluster-wide revision counter and assigns the new revision $R' > R$. Because etcd executes all write transactions (Txn) serially through a single Raft leader log and checks the precondition Key.ModRevision == R before applying the update, two concurrent readers starting from revision $R$ cannot both succeed: the first commits at $R_1 > R$, causing the second's ModRevision == R check to evaluate to False (409 Conflict).

Enabling Watch Bookmarks (allowWatchBookmarks=true) reduces 410 Gone full re-LIST storms after etcd compaction by > 98%, keeping kube-apiserver memory flat on 5,000-node clusters. In production systems (etcd v3 (mvcc treeIndex + bbolt B+ Tree), kube-apiserver cacher (WatchCache ring buffer), and Kubernetes Server-Side Apply field-ownership tracking.), engineers monitor these exact trade-offs to keep p99 latency predictable and prevent resource exhaustion under peak load.

What the interviewer is listening for

• Traces how etcd MVCC Revisions, Kubernetes metadata.resourceVersion, 409 Conflict Optimistic Concurrency, and Watch Bookmarks operates in memory step by step rather than giving a vague textbook definition.
• States the core correctness invariant and explains why edge cases do not violate it.
• Quantifies time, space, and hardware costs (etcd Raft Write (PUT / PATCH), kube-apiserver Watch Cache Hit) with concrete numbers.
• Identifies real-world failure modes (Writing a controller loop that calls clientset.CoreV1().Pods("").List(ctx, metav1.ListOptions{}) every 5 seconds instead of using a SharedInformer) and explains how production systems mitigate them.

Common mistakes

• Treating etcd MVCC Revisions, Kubernetes metadata.resourceVersion, 409 Conflict Optimistic Concurrency, and Watch Bookmarks as a black box without explaining the underlying memory or state transitions.
• Ignoring worst-case tail latency or memory overhead (Retrying a 409 Conflict on PUT by blindly overwriting the server's resourceVersion on your stale in-memory struct and re-sending PUT!).
• Applying the pattern without enforcing explicit capacity bounds or validation (Placing etcd's data directory on shared network block storage alongside noisy build/logging disks).