The happy path of dns-lookup-trace ends in the resolver's cache. This trace is the bad day: every cache is empty, and the full machinery of the DNS hierarchy has to run. The striking thing is how much of the internet a single cold name touches — and how cheap the warm version of this trace is.
Three caches miss in sequence. The libc resolver has no record; the local resolver (systemd-resolved or your router) has no record; and the recursive resolver at your ISP or 8.8.8.8 has no record either. Each miss is detected by a hash-table lookup that costs microseconds. The first packet hasn't flown yet, but the lookup chain that will cost 50–150 ms has started.
The stub sends its UDP query for example.com to the recursive resolver. The resolver checks its own cache, misses, and takes over the walk. This is the trust boundary: your machine has delegated the entire hierarchy to one server it trusts to recurse on its behalf. The resolver now faces the same problem — it needs com's nameservers to ask for example.com.
The resolver asks one of 13 root servers (anycasted, so it's likely a nearby instance) for com's nameservers. The root answers with a referral: a list of com TLD servers plus their glue addresses — not the final answer, just a pointer. The resolver repeats the pattern against a TLD server: "where is example.com?" → referral to the authoritative nameservers (often four of them, a.iana-servers.net and friends). Each step is a fresh UDP exchange of roughly 1–10 ms on a good network path.
The resolver queries an authoritative server for example.com's A record. This time the answer is final: 93.184.216.34, TTL 3600, served directly from the zone file — no referrals, no further indirection. The resolver returns the answer to your stub, which returns it to getaddrinfo, which returns it to the application. Total cold-path wall clock: typically 50–150 ms, almost all of it network RTTs, of which the last hop alone is the one that mattered.
Now the TTLs that rode along in every answer decide the future. The answer record (TTL 3600) lives in the recursive resolver's cache for an hour; the referrals (root hints, TLD lists) are cached for days. Your libc cache holds the answer for 60 seconds by default (timeout:nss semantics). The next lookup for this name costs ~100 µs, not 100 ms — the entire trace collapses to a hash-table hit because the resolver remembered the hard part.
dig +trace example.comWhat the machine actually does on a cold lookup is walk a delegation chain that was designed in 1983: three levels of indirection, each answering only "ask this server instead," until one server actually owns the name. The cost is a stack of RTTs — and the entire discipline of DNS engineering is deciding who caches that stack's result and for how long.