The notification service delivers messages across three channels — push (APNs/FCM), SMS (Twilio etc.), email (SES) — from a single API. At 50M notifications/day the machine sustains ~580/sec average and 5K/sec during campaign bursts, to tens of millions of registered devices.
The API tier validates, looks up the user's channel preferences, and enqueues one notification per channel — never inlining provider calls into the request path. Each channel has its own queue and its own worker pool, because providers have wildly different limits: FCM accepts tens of thousands of pushes/sec, SMS vendors throttle to hundreds/sec, email is elastic. Channel-specific workers also isolate failure: an SMS provider outage stalls only SMS.
Provider retries live in the worker, not the API: exponential backoff (1s, 4s, 16s) up to 3 attempts, then dead-letter with alerting. The machine tracks per-provider rate limits with a token bucket and shapes send rate to vendor quotas rather than hammering and eating 429s.
Dedup is critical — retries and duplicate events must not double-send. Every notification carries an event_id; the machine writes it to Redis with a 24h TTL before the first enqueue, and a duplicate event_id is dropped. Preference and opt-out checks happen at ingest: the service reads the user profile and drops the channel before queueing, so dead messages never occupy queues.
Data model: notifications(id, event_id, user_id, channels, payload, created_at), per-channel queue entries, delivery_log(id, channel, status, attempt, provider_id). The delivery log is the source of truth for observability — per-channel success rates and latency are computed from it.
Bottlenecks: provider latency (an SMS send takes 1-5s — workers must be async, not thread-per-send); the dedup Redis key growth (expire aggressively); and campaign fanout hitting the preference lookup hot path (cache user prefs).
API → validate → prefs/opt-out check → dedup (Redis event_id)
→ push queue → push workers → FCM/APNs
→ sms queue → sms workers → SMS vendor (token-bucket shaped)
→ email queue→ email workers→ SES
→ delivery_log ← all statuses; dead-letter → alerting