The chat system must deliver messages between users with low latency, exactly-once-enough ordering, and delivery to offline users on reconnect. Functional requirements: 1:1 and group chats, message history, presence, read receipts, and push notifications. At 100M DAU with maybe 1M concurrently connected, the machine holds ~1M open WebSocket connections and moves ~1M messages/sec at peak typing rates.
Connections are the first bottleneck: no single box holds a million sockets. The machine runs gateway clusters of ~100K connections each, fronted by an L4 load balancer that pins a connection to one gateway for its lifetime (sticky by connection id). Gateways only move bytes; they do not store state.
Messages flow through a two-stage pipeline. The sender's gateway accepts the message, assigns a monotonically increasing sequence number per conversation (a Redis INCR or per-conversation log), appends to the conversation log (Cassandra or Kafka), and publishes to a fanout topic. Delivery: each member has an inbox queue; the fanout worker pushes the message id into every member's queue — for a 100K-member group that is 100K queue writes per message. Online members get a push over their gateway socket; offline members catch up on reconnect by pulling their inbox after their last sequence.
The data model: messages(conversation_id, seq, sender, body, ts) keyed for range scans, inboxes(user_id → set of msg ids), and conversations(members). Ordering is per-conversation only — cross-conversation order is meaningless.
Bottlenecks: fanout writes for giant groups (mitigate with per-group delivery sharding and offline batching); gateway rebalancing on scale events (drain gracefully); clock-free ordering — the machine must never trust client timestamps, only the log sequence.
Sender → Gateway LB → Gateway A → seq (Redis) → Conversation log (Kafka/Cassandra)
↓ fanout worker
Member inbox queues → online: Gateway B → socket
→ offline: catch-up on reconnect