Exactly-Once Is a Lie or at Least Not the Guarantee You Think It Is

I work mostly on systems where duplicate side effects aren't harmless. If a notification is sent twice, it's annoying. If a wallet credit, withdrawal, or bet settlement happens twice, you've created a financial problem. That's why I've become fairly conservative about exactly once claims.

The title is intentionally provocative. Exactly once semantics can exist inside a defined transactional boundary: a broker may deduplicate a publish, and a stream processor may atomically commit its output and offsets. The mistake is assuming that broker level exactly once processing also makes an independent database, wallet, payment provider, or HTTP API apply a business effect exactly once.

Exactly-once processing semantics are not exactly-once business effects across arbitrary distributed boundaries. The useful engineering question is therefore not only “what does my broker guarantee?” It is: if this event is observed more than once, can the application prove that the business operation is applied once? That shifts correctness to the boundary that owns it: the service and database changing the business state.

1. The $100 double credit problem

Wallet and gaming systems make this especially painful. Imagine a settlement event arrives after a winning round. We credit the player's wallet and commit the transaction, but the consumer crashes before acknowledging the message. The broker hasn't done anything wrong when it redelivers the event. In fact, that's exactly what I want it to do. The problem is on my side if the wallet service interprets “I received this message again” as “credit the player again.” This is where exactly once stops being just a messaging discussion. The real question becomes: how do I guarantee that one business operation produces one business effect, regardless of how many times the message is delivered? Consider a settlement workflow. A bet is resolved and a BetSettled event instructs the wallet service to credit a player with $100. The wallet transaction commits successfully. Before the consumer acknowledges the message, the process crashes. The broker never sees the acknowledgement, so it redelivers the event to the restarted consumer.

Redelivery is not a broker defect. It is the only safe response when the broker cannot distinguish “the consumer failed before committing” from “the consumer committed and failed before acknowledging.” This is why I don't treat retries as an error condition. In most of the systems I've worked on, retries are expected. The dangerous part is when the business operation wasn't designed to survive them. A payment, bet settlement, or wallet credit should be safe even if I process the same message twice.

2. Delivery guarantees, without marketing language

RabbitMQ, Kafka, Azure Service Bus, SQS style queues, and similar systems expose different terminology and mechanics, but the application boundary sees the same fundamental uncertainty: network calls can time out after succeeding, acknowledgements can be lost, consumers can restart, leases can expire, and retry policies can deliver the same logical message again.

Even when a Kafka pipeline uses transactional producers and exactly-once stream processing, a consumer that calls a separate PostgreSQL wallet or payment provider has crossed the guarantee boundary. The remote side effect is not atomically committed with the broker offset unless the application introduces its own coordination—and a distributed transaction across arbitrary systems is normally the wrong tool.

3. Why redelivery is normal

• Consumer crash: business data commits, then the process exits before acknowledging.
• Lost acknowledgement: the ACK is sent but the connection fails before the broker records it.
• Visibility timeout or lock expiry: processing takes longer than the queue lease, so another worker receives the message.
• Network timeout: the caller cannot tell whether a remote operation failed or completed after the response was lost.
• Producer retry: publishing succeeds, but the confirmation is lost and the producer publishes again.
• Operational replay: engineers deliberately replay a topic, restore a queue, or rebuild a projection.

These are ordinary failure modes, not edge cases. A financial workflow cannot “retry and hope” because the cost of a false retry is money. It needs a stable identity for the business operation and a durable record that says whether that identity has already changed state.

4. Business-level idempotency

An idempotent operation produces the same business outcome when repeated with the same identity. The identity must describe the operation, not the transport attempt. A new delivery tag, queue receipt, or retry counter is not an idempotency key because it changes on every attempt. “Have I processed this message?” is useful, but for financial workflows the more important question is often: “Has this business operation already happened?”

Four identifiers commonly appear in this flow, and they solve different problems:

• Event ID: an immutable unique ID assigned when the event is created. It deduplicates the event envelope.
• Message ID: the broker envelope or publish identity. It helps trace and deduplicate a particular publication, but republishing the same domain fact may create another message ID.
• Correlation ID: the identity of the wider request or workflow. It connects logs and events; it is usually too broad to be a deduplication key.
• Business idempotency key: a stable key for the domain effect, such as settlement:wallet_42:round_91:player_7:win or refund:payment_321:v1. It prevents two different events from applying the same business operation twice.

The distinction matters. A buggy producer might publish two different event IDs for the same settlement. Event level deduplication would accept both; a unique key built from the invariant—such as wallet + round + player + settlementType—would reject the second credit. The business owns that boundary, not the broker.

type SettlementEvent = {
  eventId: string;          // immutable transport identity
  betId: string;            // aggregate identity
  settlementId: string;     // business idempotency identity
  aggregateVersion: number;
  amountCents: number;
};

5. The idempotent consumer and inbox pattern

The simplest safe consumer writes an inbox record and performs the business update in the same local database transaction. A unique constraint wins races between concurrent deliveries. Application checks alone are insufficient under concurrency:

Worker A -> SELECT says settlement does not exist
Worker B -> SELECT says settlement does not exist
Worker A -> INSERT wallet credit
Worker B -> INSERT wallet credit

Without a unique business constraint, both workers can make a locally reasonable decision and still violate the system invariant. The database constraint is not merely validation; it is part of the distributed systems correctness model.

async function handleSettlement(event: SettlementEvent): Promise<void> {
  await database.transaction(async (tx) => {
    const accepted = await tx.inbox.insertIfAbsent({
      consumer: "wallet.settlement.v1",
      eventId: event.eventId,
      receivedAt: new Date(),
    });

    if (!accepted) return; // a previous delivery already committed

    await tx.walletCredits.insert({
      settlementId: event.settlementId, // UNIQUE business key
      walletId: await tx.wallets.requireForBet(event.betId),
      amountCents: event.amountCents,
    });

    await tx.wallets.incrementBalanceForSettlement(
      event.settlementId,
      event.amountCents,
    );
  });
}

In PostgreSQL, the inbox can use UNIQUE (consumer, event_id), while the credit ledger uses UNIQUE (settlement_id). In MongoDB, equivalent unique compound indexes can enforce the same invariants, and a transaction can include the inbox insert plus wallet ledger write when both collections share the transaction boundary.

The acknowledgement happens only after the transaction commits. If the transaction rolls back, the message should remain retryable. If the commit succeeds and the acknowledgement is lost, the next delivery hits the unique inbox record and becomes a cheap no-op.

6. Scenario: consumer crashes after database commit

T0  Bet settles
T1  Consumer starts the database transaction
T2  Wallet credit + Inbox record commit
T3  Consumer attempts ACK
T4  Network connection drops
T5  Broker redelivers the settlement event
T6  Consumer receives the same settlement again
T7  Unique business key detects the previous settlement
T8  Consumer ACKs safely

The duplicate message is not the failure; it is a normal recovery action after an uncertain acknowledgement. Crediting the wallet twice would be the failure. The consumer is free to execute more than once because the business transition cannot commit more than once.

7. The dangerous producer dual write

Consumer idempotency protects the receiving side, but the producer has a different problem. A naive service performs two independent writes:

1. Update the bet to SETTLED in the database.
2. Publish BetSettled to the broker.

If the database succeeds and publishing fails, the bet is settled but the wallet never learns about it. If publishing succeeds and the database transaction rolls back, downstream services act on a settlement that does not exist. Reversing the order only reverses which inconsistency is possible.

A retry does not resolve the uncertainty. After a timeout, the producer often cannot know whether the broker accepted the event. Blindly retrying may create duplicates; refusing to retry may lose the event.

8. Transactional outbox

The transactional outbox turns the business update and the intent to publish into one local atomic operation. The service updates the aggregate and inserts an outbox row in the same database transaction. A separate relay publishes pending rows and marks them sent. If publishing or confirmation fails, the relay retries.

await database.transaction(async (tx) => {
  const bet = await tx.bets.settle({
    betId: command.betId,
    expectedVersion: command.expectedVersion,
  });

  await tx.outbox.insert({
    eventId: crypto.randomUUID(),
    aggregateId: bet.id,
    aggregateVersion: bet.version,
    eventType: "BetSettled",
    payload: serializeSettlement(bet),
    occurredAt: new Date(),
  });
});

The relay should claim rows safely, publish with confirms where the broker supports them, record attempts, and use backoff. It must assume that it can crash after publish but before marking the row sent. That is why the event ID is persisted before publication and reused on every attempt.

9. Scenario: producer updates the database but publish fails

With a naive dual write, the settled bet and absent event form a permanently inconsistent state unless a reconciliation job discovers it. With an outbox, the committed transaction contains both the new bet state and a pending event. The relay can be down for minutes without losing the intent. When it recovers, the outbox backlog drains and downstream state converges.

This is eventual consistency with an explicit recovery path, not wishful thinking. The business transaction is immediately consistent inside its service boundary; cross service propagation is asynchronous and observable.

10. Scenario: duplicate settlement event

Suppose a producer bug creates two envelopes with different event IDs for the same settlement. The inbox sees two new events, so transport deduplication alone is insufficient. The wallet ledger must also enforce the business invariant:

CREATE UNIQUE INDEX wallet_credit_once
  ON wallet_credits (settlement_id);

CREATE UNIQUE INDEX inbox_event_once
  ON consumer_inbox (consumer_name, event_id);

The database constraint is the final concurrency guard. Application checks improve error messages and avoid unnecessary work, but correctness should not depend on two competing workers observing state in the same order. Treat the expected unique-key conflict as an idempotent outcome, then verify that the stored operation matches the incoming amount, currency, and target; the same key with different business data is a conflict to investigate, not a silent success.

11. Retries, poison messages, and dead letter queues

Retries should distinguish transient failures from permanent ones. A database timeout or temporary dependency outage may succeed later. An invalid currency, impossible state transition, or schema mismatch will not be repaired by immediate repetition.

• Transient: retry with exponential backoff, jitter, and a bounded attempt count.
• Rate limited: respect provider retry hints and isolate concurrency by dependency.
• Permanent business rejection: record the rejected transition and stop retrying.
• Poison message: move it to a dead letter queue with error category, event ID, correlation ID, attempt count, and sanitized diagnostics.

A DLQ is not a graveyard. It needs ownership, alerts, a replay procedure, and tooling that preserves the original event ID. Replaying with a new identity bypasses the very deduplication safeguards intended to make replay safe.

12. Scenario: out of order events

Idempotency handles repetition, not ordering. A service may observe BetPlaced, BetCancelled, and BetSettled in an unexpected order because partitions, retries, multiple publishers, or parallel consumers changed arrival timing.

Sensitive aggregates need explicit transition rules:

• Aggregate version: accept version 7 only after version 6; buffer briefly or request reconciliation when a gap appears.
• State-machine validation: reject SETTLED → CANCELLED unless the domain defines a compensating transition.
• Partitioning: route one aggregate key to one ordered partition where available, while remembering that retries can still duplicate.
• Optimistic concurrency: update with WHERE version = expected_version; a zero row update signals stale work.
• Sequence numbers: compare a producer owned monotonic sequence, not local consumer time.

Do not blindly apply an older event because it arrived later. Do not blindly discard it either: a gap may represent a lost event that requires replay or state reconciliation. The correct response depends on whether the consumer owns a projection, a financial ledger, or an authoritative aggregate.

13. Observability is part of correctness

A system is not reliable merely because duplicates are ignored. It must make abnormal delivery behavior visible before an outbox backlog, retry storm, or poison message turns into a customer-facing incident.

Structured logs should include event ID, business idempotency key, aggregate ID, aggregate version, correlation ID, causation ID, consumer name, delivery attempt, and outcome. They should not include reusable credentials, full payment payloads, or unnecessary personal data.

14. Reconciliation is the last safety net

Inbox, outbox, idempotency, and retries make known failure modes recoverable. They do not prove that every dependency, migration, operator action, or latent bug behaved as expected. Financial and settlement systems still need reconciliation: an independent process that compares authoritative records and reports impossible or incomplete combinations.

• Compare wallet ledger entries with bet or round settlement records.
• Compare payment provider transactions with internal payment and refund state.
• Detect committed outbox records that remain unpublished beyond their service level objective.
• Detect inbox events marked processed without their expected downstream ledger entry or state transition.
• Detect round settlements with no matching player ledger entries, or ledger entries with no valid settlement.

Reconciliation should produce actionable exceptions with an owner, evidence, and a safe repair path; it should not silently rewrite financial state. It is the final safety net when the model encounters a failure nobody predicted, and its independence from the main event path is exactly what makes it valuable.

15. System boundaries

Precise boundaries lead to better contracts. “Exactly once” is too vague unless the team can name the operation, storage boundary, deduplication identity, retention window, and recovery procedure.

16. Stronger guarantees have a cost

Inbox, outbox, DLQ, sequencing, and reconciliation are not free reliability switches. They introduce additional database writes, larger storage requirements, inbox and outbox cleanup policies, operational monitoring, DLQ ownership, replay tooling, reconciliation jobs, more complicated debugging, and potential throughput impact. They also increase the number of states an operator must understand during an incident.

Those costs are justified when an incorrect outcome costs more than the architecture needed to prevent and repair it. A duplicated analytics event may slightly distort a chart and can often be deduplicated later. A duplicated $100 wallet credit changes money and audit state immediately. The first may need a simple consumer; the second needs a durable invariant.

17. When you should not use all of this

Reliability requirements should follow the cost of duplication, loss, inconsistency, and recovery. Applying a financial-grade inbox, ledger, DLQ, ordering protocol, and reconciliation loop to every cache or telemetry event creates operational weight without buying meaningful business safety. Reliability is a business requirement before it is an infrastructure requirement.

Analytics duplicates may be accepted or removed asynchronously. Cache invalidation is naturally replay safe. Notifications usually need a lightweight send key, not a complete financial workflow. Webhooks start with the provider event ID, then add a business key if the handler moves money or inventory. Wallet credits, payments, refunds, and bet or game settlement need stronger invariants, durable audit records, retry safety, and—where state transitions are order-sensitive—version checks.

18. What I would do for financial workflows

1. Assign every event a unique immutable event ID at creation time.
2. Assign every financial operation a domain owned idempotency key.
3. Persist business state and the outbox record in one local transaction.
4. Use a consumer inbox plus unique business constraints at the receiving service.
5. Acknowledge only after the consumer transaction commits.
6. Make retries bounded, classified, delayed, and measurable.
7. Monitor DLQs and define an audited replay procedure.
8. Carry correlation and causation IDs through every event.
9. Use structured logs and metrics for duplicates, retries, lag, and backlogs.
10. Validate state transitions and versions for sensitive aggregates.
11. Run reconciliation jobs between authoritative ledgers and derived state.
12. Never trust broker delivery semantics alone for financial correctness.

Retention also needs a decision. An inbox kept forever grows without bound; an inbox deleted too early permits an old replay to repeat a business effect. Financial ledgers can usually keep the business key permanently while transport inbox rows follow a documented replay horizon and archival policy.

19. Final takeaways

Reliable event-driven systems are not built by trying to eliminate retries or duplicates. They assume that messages can be duplicated, consumers can crash, networks can fail, events can arrive late, acknowledgements can disappear, and services can temporarily disagree.

• Give each business operation a stable, domain-owned identity.
• Enforce the invariant in the same transaction as the state change.
• Make publication, retries, replay, and reconciliation durable and observable where the risk justifies them.
• State the exact boundary of every delivery or processing guarantee.

Transport-level exactly-once semantics can reduce duplicate processing inside their defined boundary. Business-level idempotency prevents duplicate money after the workflow crosses that boundary. Do not design the transport to behave perfectly. Design the business operation to remain correct when the transport does not.

Continue with Designing Real-World Systems, the CAP theorem trade-offs, or the multi-chain payment engine case study for related architecture decisions.