Notifications are push (APNs/FCM), email, and SMS, and the design is the same regardless of channel: producers never call channels. The queue is the contract — producers publish and move on; the notification service owns delivery.
Requirements and capacity
100M notifications/day ≈ 1.2k/s average, 5–10x peak during campaigns. SLA: at-least-once delivery, p99 enqueue-to-provider <30s. Retention: 100M/day × ~200B × 365 ≈ 7.3TB/year of notification records — partition by month, prune the rest.
Shape
Event → API/consumer → persist → per-channel queue → provider adapter (rate-limited, circuit-broken) → provider. The per-channel split is not cosmetic: an SMS outage must not block email. Each channel gets its own workers, backoff policy, and dead-letter queue.
Providers are the bottleneck
APNs and FCM throttle bursts; email providers cap per-sender rates; SMS is cost-bound. The adapter holds a token bucket per provider — the rate-limit math applies here, but the budget is the provider's capacity, not your policy. A 9am campaign that produces 40M emails does not change the provider's sustainable rate: the queue absorbs the burst (queue-burst-trace) and workers drain at the provider's pace, not the campaign's. Queue depth is the safety valve — name it, and what alerts on it.
Exactly-once is impossible; bounded duplicates are the goal
Dedup window (5 minutes) on (user_id, channel, template_id, correlation_id) enforced by a unique index or Redis SETNX: the first insert wins, the retry returns the existing notification. At-least-once delivery plus a dedup window is the honest contract.
Data model
notifications(id, user_id, channel, template_id, payload, status, attempt_count, next_retry_at, provider_message_id, dedup_key UNIQUE) — index on (user_id, created_at) for the history view; status transitions pending → sent | failed → dead. Check preferences (opt-outs) before enqueue, from cache — a user who opted out should never be a queue entry at all.
Retries
Exponential backoff with jitter, cap at ~5–8 attempts, then the DLQ. The subtle failure: retries re-enter the same rate-limited path, so a provider outage turns into a retry storm that amplifies itself — the circuit breaker (open on provider error rate) is what stops it (webhook-retry-trace). When the breaker opens, the queue stops draining that channel; the DLQ is where the failures are visible.
Bottlenecks
Provider capacity and cost (SMS especially), campaign bursts (pre-warm queues, stagger sends), provider degradation (breaker + degrade to a lower-cost channel), and the dedup index under producer retry storms. The service itself is embarrassingly parallel — the queue, the rate limit, and the breaker are the design.