Chapter 4 of 8

Make effects safe under duplicate and ambiguous delivery

Chapter 4

About 5 minutes · includes examples, an exercise, and references

Chapter at a glance

•Assign one stable idempotency key to one canonical logical intent and reject semantic reuse.
•Represent dispatch uncertainty explicitly and reconcile against an authoritative source before retrying.
•Use outbox and inbox or effect-ledger patterns to make at-least-once delivery inspectable.

External effects are where loop correctness meets irreversible reality. Networks provide uncertainty: a request can reach a service, commit, and lose its response. The caller sees a timeout but cannot infer whether nothing happened or the effect succeeded. Retrying may be necessary for availability and dangerous for correctness. Loop Engineering handles this by combining durable intent, idempotent commands, receipts, reconciliation, and compensation.

RFC 9110 distinguishes safe methods, whose requested semantics are read-only, from idempotent methods, where multiple identical requests have the same intended effect as one. The property belongs to semantics, not a verb's spelling. A POST can support idempotent creation when the API defines a caller-provided request identifier; a nominally idempotent update can still trigger duplicate emails if the implementation hides non-idempotent side effects. Document all durable effects, including notifications, audit entries, metering, and downstream events.

An idempotency key identifies one logical intent across delivery attempts. Generate it at the authority boundary, before dispatch, and persist it with canonical request parameters. On the receiver, atomically reserve the key and semantic request, execute or retrieve the stored outcome, and return the same logical result for duplicates. If the same key arrives with materially different parameters, reject it. Amazon's guidance stresses caller request identifiers because the server cannot reliably infer whether two similar calls represent a retry or two desired resources.

Scope and lifetime matter. A key may be unique per tenant and operation, not globally. Its retention must exceed the maximum retry, redelivery, and delayed-message window. Expiring it too early can turn a late retry into a second effect. Retaining it forever can be expensive or conflict with data policy. Store a payload digest, status, result reference, and timestamps; protect access because results may be sensitive.

Exactly-once delivery across independent systems is generally the wrong mental model. Aim for at-least-once delivery plus idempotent application, or at-most-once attempts plus reconciliation depending on risk. A transactional outbox atomically stores domain change and message intent, but consumers still see duplicates. An inbox or effect ledger records consumed message identity. These patterns move uncertainty into durable, inspectable state rather than eliminating it.

Use an effect lifecycle: intended, dispatched, confirmed, rejected, or ambiguous. Confirmation requires a receipt or authoritative observation. A timeout after dispatch enters ambiguous, not rejected. The reconciliation action queries the destination by idempotency key or business identifier and classifies the outcome. If found with matching semantics, record confirmed. If definitively absent and retry remains allowed, redispatch the same intent. If conflicting or unknowable, escalate. Never create a new key merely because the first attempt timed out.

Read-after-write consistency affects verification. The destination may commit before a replica or search index exposes the record. Reconciliation needs an authoritative read or a bounded stabilization period. The loop must distinguish “not visible yet” from “definitively absent.” Encode the consistency assumption, maximum visibility delay, and the source of truth. Otherwise a fast verification loop can manufacture duplicates from a normal propagation delay.

Some effects cannot be made idempotent at the destination. Wrap them with a local effect ledger and a unique business constraint where possible, narrow retry policy, and prefer human reconciliation for ambiguous high-impact actions. For email, a duplicate may be tolerable and measurable; for a wire transfer, the same uncertainty demands stronger provider semantics and operational review. Reliability policy follows consequence, not implementation convenience.

Compensation is not rollback. The Saga pattern decomposes a long-lived transaction into subtransactions with compensating actions. A refund compensates a captured charge but does not erase the fact that the charge occurred; a cancellation email cannot be unsent; a published secret may already be copied. Design compensations per effect, record their own idempotency keys and receipts, and consider whether they require approval. A saga can itself fail and needs a terminal needs_operator state.

Avoid dual writes such as updating a database and separately publishing a queue message without a recovery record. If the database commits and publish fails, the loop loses work; if publish succeeds and the database rolls back, consumers see a phantom. Use a local transaction with outbox, change-data capture, or a workflow engine whose durable command record provides equivalent recovery. The choice must be matched to failure and ordering requirements.

The worked proof for an effect is not “the API returned 200.” It is a chain: an authorized canonical intent with a stable identity; a receiver guarantee for duplicate semantics; a dispatch record; an authoritative receipt or reconciliation result; invariant checks on cardinality and scope; and, where necessary, a compensation plan. Once this evidence is explicit, retries become a controlled transition rather than hopeful repetition.

Key points

Assign one stable idempotency key to one canonical logical intent and reject semantic reuse.
Represent dispatch uncertainty explicitly and reconcile against an authoritative source before retrying.
Use outbox and inbox or effect-ledger patterns to make at-least-once delivery inspectable.
Treat compensation as a new forward effect with its own failure, approval, idempotency, and evidence.

An in-memory idempotency ledger with semantic checks

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

An in-memory idempotency ledger with semantic checkspython

from dataclasses import dataclass
from hashlib import sha256
import json

@dataclass(frozen=True)
class Record:
    payload_digest: str
    result: dict

ledger: dict[tuple[str, str], Record] = {}

def canonical_digest(payload: dict) -> str:
    raw = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return sha256(raw).hexdigest()

def create_charge(tenant: str, key: str, payload: dict) -> dict:
    identity = (tenant, key)
    digest = canonical_digest(payload)
    prior = ledger.get(identity)
    if prior:
        if prior.payload_digest != digest:
            raise ValueError("idempotency key reused with different semantics")
        return prior.result
    # A real service performs reservation and effect in one transaction.
    result = {"charge_id": f"ch-{len(ledger) + 1}", "status": "committed"}
    ledger[identity] = Record(digest, result)
    return result

payload = {"order": "O-9", "amount": 2500, "currency": "USD"}
first = create_charge("tenant-a", "intent-77", payload)
second = create_charge("tenant-a", "intent-77", payload)
assert first == second and len(ledger) == 1
print(first)

Worked examples

Toy

Duplicate-safe points award

A queue delivers the same award-points message twice.

The consumer transaction inserts message identity into an inbox table with a uniqueness constraint and increments the balance only on the first insert. Duplicate delivery returns the stored outcome. Reusing the identity with a different amount is rejected and alerted.

Deduplication and balance update are atomic.
Semantic mismatch is not treated as a retry.
Duplicate delivery produces the original outcome.

Application

Payment timeout after commit

A payment provider commits a charge but the response is lost.

The workflow records ambiguous after the deadline. It queries the provider using the original idempotency key. Finding a matching charge produces confirmed and stores the provider receipt. Finding a conflicting amount produces needs_operator. Only a definitive absent result permits redispatch with the same key.

Timeout is not classified as failure.
The original key survives retries.
Authoritative lookup precedes any repeat effect.

System

Order saga with failed compensation

An order reserves inventory, charges payment, then fails shipment creation; releasing inventory succeeds but refunding payment times out.

Each forward and compensating action has its own stable key and receipt. Refund becomes ambiguous and is reconciled against the payment ledger. The order remains compensating, not cancelled, until both compensations are confirmed. Exhaustion raises an operator case with all effect identities.

Compensation does not erase the forward history.
Terminal status reflects all confirmed effects.
Operator evidence identifies every outstanding obligation.

Exercise

Design an effect protocol for the worst timeout

Choose a consequential external write and specify behavior for duplicate delivery and loss of every response.

Define canonical intent, key scope, semantic-mismatch rule, retention window, and receiver transaction.
Draw intended, dispatched, confirmed, rejected, and ambiguous transitions.
Specify an authoritative reconciliation query and consistency assumptions.
Design compensation, including its own idempotency and failed-compensation terminal state.

Success criteria

No timeout path creates a new logical intent accidentally.
Duplicates with matching parameters return one logical result.
Late visibility cannot be mistaken for definitive absence.
The workflow never claims a compensated terminal state without receipts.

Reflect: Which hidden secondary effect would violate idempotency even if the main database row is unique?