The crawler must fetch pages at scale without breaking the sites it visits. At 1B pages re-crawled monthly, the machine fetches ~385 pages/sec sustained — ~4M fetches/day — while respecting each domain's robots.txt and load tolerance.
The pipeline: a frontier of pending URLs feeds a fetcher pool; each fetch produces raw HTML; a parser extracts links and page content; extracted links pass through dedup before re-entering the frontier; page data lands in object storage with metadata in a DB.
Politeness is the design constraint that shapes everything. Crawling is polite if the machine sends at most one request per domain per second (tunable). The frontier is therefore not one queue but a set of per-domain queues, and a scheduler drains each domain queue at its allowed rate. Fetchers never pick a URL directly; they pull from the domain scheduler, which enforces the token bucket per domain. The DNS layer must cache resolutions — a naive crawler spends 30% of its time on DNS.
Dedup uses two structures: a persistent urls_seen set (bloom filter over the URL hash, backed by a KV store for exactness) for discovered URLs, and per-crawl state in a KV store. URLs re-enter the frontier only for scheduled re-crawl. The machine must also canonicalize: strip fragments, resolve redirects, normalize scheme — http://A.com and https://a.com/ are the same page to a search index.
Storage: raw HTML in object storage (S3), pages(url_hash, url, domain, fetch_time, http_status, content_hash) in the DB, and a crawl_queue per domain in Redis. Content-hash dedup catches near-identical pages without fetching.
Bottlenecks: DNS (cache aggressively); slow domains pinning fetcher slots (bound wait time, skip to next domain); infinite link cycles (depth limit and URL-set dedup); robots.txt fetches (cache with TTL and re-check on re-crawl). Failure handling: transient errors retry with backoff, hard 404s drop the URL, five consecutive failures park the URL in a quarantine list.
Seed URLs → frontier (per-domain queues)
→ domain scheduler (token bucket, robots.txt check) → fetcher pool
→ parser → links → dedup → frontier
→ raw HTML → object storage; meta → DB; content_hash dedup