The trap here is "plenty of CPU" — the interviewer wants you to enumerate every queue and lock between the NIC and the response, because latency degrades long before utilization hits 100%. This is a Little's Law question dressed up as a performance question.
The mental model: latency = time in queue + time in service, and queues form everywhere.
Start with Little's Law: L = λW. At 1000 req/s with 200ms average response time, you need ~200 requests in flight. All 200 are simultaneously contending for: socket buffers, thread-pool slots, DB connections, and locks. As concurrency climbs, time-in-service stays flat but time-in-queue grows superlinearly. The system is "loaded" well before the CPU is.
Where the queues form:
- Threads. A thread-per-request model with a 200-thread pool: at 200 in-flight requests, the queue is full — request 201 waits for a thread, and its wait time adds directly to observed latency. Even with an async runtime, the task scheduler's run queues and its timers add scheduling latency when saturated.
- Kernel sockets. Each connection owns receive and send buffers. With many in-flight requests, the kernel's receive queue and the epoll ready lists grow; the TCP stack's softirq processing (see the receive path) contends for the same cores the app runs on — at ~1GB/s of copy work at line rate, memory bandwidth and cache-line contention become the ceiling before core count does.
- Locks and shared state. A single shared counter, an in-memory cache with a mutex, a connection pool with a contended semaphore: lock acquire times triple or quadruple under saturation as cache lines ping-pong between cores. A lock that takes 1µs uncontended can take 20-50µs contended — invisible in a profile at 10% utilization, dominant at 80%.
- Downstream dependencies. The API itself is fine; the DB connection pool (typically 50-100 connections) becomes the true bottleneck. Requests queue at the pool, and queue time is latency. 500ms DB queries × 50 connections caps you at ~100 req/s of DB throughput — no amount of spare CPU changes that.
- The accept/connection path. Epoll wakeups, accept() contention on the listen socket, and connection establishment under SYN backlog pressure add ms-scale waits that compound per request.
How you'd prove it: measure with utilization at the dependency (pool utilization, queue depth at the DB, softirq CPU%), then plot p50/p99 latency vs concurrent load — latency rises linearly with queue depth exactly where a pool saturates. Fixes follow the bottleneck: connection reuse and HTTP keep-alive (fewer handshakes, fewer sockets), bounded queues with rejection, pooling with accurate limits, and batching to raise service rate per connection.