Multi-channel Notification Service

The constraint

Deliver millions of messages across unreliable external providers while maintaining priority, traceability, and retry safety.

The solution

Separated message acceptance from delivery. Priority queues feed channel-specific workers, retry policy is failure-aware, and terminal failures move to dead-letter queues with full diagnostic context.

Acceptance is not delivery

An acceptance boundary should acknowledge a notification only after validating the command and recording durable delivery work. That state is accepted, not delivered. The lifecycle then moves through queued and processing before a provider receipt can confirm delivery. A rejected address or exhausted retry policy ends as a failed or terminal-failure state instead of being presented as a successful send.

Keeping these states explicit gives callers an honest contract and lets operators distinguish an ingestion problem from a provider problem. The existing delivery-state tracking supplies that foundation; richer state-transition history and reconciliation policies would be production evolutions rather than claims about the current portfolio implementation.

Retry the business operation safely

Idempotency belongs to the delivery operation, not to one RabbitMQ message. A stable key should represent the caller's notification intent, recipient, channel, and relevant template or payload version. If the queue redelivers or a worker republishes the command, that identity must resolve to the existing delivery record instead of producing another user-visible notification.

A safe retry policy separates transient failures such as timeouts or provider throttling from permanent failures such as an invalid destination. Transient failures use bounded exponential backoff; permanent failures stop immediately. Attempt limits prevent an unhealthy provider from turning the retry queues into a storm, after which unresolved work moves to the DLQ with its original identity and attempt context intact.

Priority needs a fairness policy

Priority queues protect time-sensitive traffic, but strict priority can starve ordinary notifications during a sustained high-priority burst. A production policy needs weighted consumption, reserved worker capacity, or aging so lower-priority work continues to advance without weakening the urgent path.

Backpressure belongs at the worker-provider boundary. When a provider is slower than incoming traffic, workers should cap in-flight calls, respect provider rate limits, and let queue depth absorb a bounded burst. Scaling workers blindly only increases throttling and timeout pressure; capacity decisions should consider the provider's safe concurrency and the age of queued work.

Treat unknown provider outcomes as unknown

A provider adapter boundary should keep credentials, request formats, error classification, rate limits, and receipt parsing outside the channel workflow. That makes provider choice replaceable without pretending every provider has identical timeout or delivery semantics. Failover ordering can then follow channel policy and provider health instead of leaking vendor-specific branches through the service.

The dangerous case is a provider accepting the request while the local call times out or the worker crashes before recording the response. Immediate failover or blind retry can send the same notification twice because the first provider operation may already have succeeded. The delivery remains unknown until a provider operation identifier, delivery receipt, status query, or reconciliation job resolves it.

The current project includes provider adapters, failover, delivery-state tracking, and receipts. Persisting every provider operation and automating unknown-state reconciliation is the production-hardening boundary; it should be designed explicitly rather than inferred from a queue acknowledgement.

DLQ replay is a recovery workflow

A DLQ preserves work that could not be completed safely, but replay is not equivalent to republishing every message. Before replay, the operator or recovery worker needs the original delivery identity, the last known provider operation, and the failure classification. An unknown provider outcome must be reconciled before another send is allowed; otherwise replay bypasses the same duplicate-prevention boundary as an uncontrolled retry.

The useful operational signals stay close to this lifecycle: queue depth and oldest-message age show backlog health; processing and end-to-end delivery latency show where time is spent; retry and DLQ rates expose failure amplification; provider error rate and latency show whether a dependency is degrading. Those signals guide concurrency and failover decisions without turning a successful queue publish into a false delivery metric.