Idempotency And Retry Safety

---
name: idempotency-and-retry-safety
description: Use when the user asks to "make this endpoint safe to retry", "add an idempotency key", "handle duplicate webhook delivery", "design a payment/booking API", "what happens if this request is sent twice", or is building any endpoint or message consumer that a client, queue, or upstream system might call more than once for the same logical operation. Guides designing idempotency keys and safe retry semantics under at-least-once delivery.
license: Apache-2.0
metadata:
  version: "0.1.0"
---

# Idempotency and Retry Safety

Distributed systems cannot cheaply guarantee exactly-once delivery — a
client that times out waiting for a response genuinely cannot tell whether
the server never received the request or received it and crashed before
replying. The practical, achievable guarantee is **at-least-once delivery
plus idempotent processing on the receiving end**, which together behave
like exactly-once from the caller's point of view.

## At-least-once vs. exactly-once: why this matters

- **At-most-once**: send once, never retry. Simple, but any dropped
  request or timeout is a silent failure — unacceptable for anything that
  charges a card or books a room.
- **Exactly-once delivery**, as a network-level guarantee, is not
  achievable in general in an asynchronous network with possible partial
  failures (the client cannot distinguish "server never got it" from
  "server got it, processed it, and the ack was lost") — treat any vendor
  claiming true "exactly-once" as actually meaning "at-least-once delivery
  with idempotent processing," which is the real, achievable target.
- **At-least-once + idempotency** is the standard, achievable pattern:
  the client retries freely on any ambiguous failure (timeout, 5xx,
  connection reset), and the server is designed so that processing the
  same logical request N times has the same effect as processing it once.

## Designing an idempotency key

1. **The key identifies one logical operation, generated by the client**,
   not the server — a UUID created once when the user clicks "book" and
   attached to every retry of that same booking attempt. If the server
   generates the key, retries of a request that never reached the server
   can't reuse it, defeating the purpose.
2. **The key's scope must be unambiguous**: usually
   `(idempotency_key, endpoint)` or `(idempotency_key, account_id)` — a key
   that's only unique per-client but collides across unrelated accounts is
   a data-leak risk, not just a bug.
3. **Store the key with the *result*, not just a "seen it" flag.** On a
   duplicate request, replay the original stored response (same status
   code, same body) — don't just skip the operation and return a bare 200,
   which changes behavior from the client's perspective between the first
   and second attempt.
4. **Set a retention window** for stored idempotency keys (commonly
   24 hours) and reject key reuse with a *different* request payload
   within that window as a client error — reusing a key with different
   parameters usually indicates a client bug, not a legitimate retry.
5. **Make the storage write and the business-effect write atomic** — in
   Postgres, insert the idempotency-key row and perform the operation
   inside the same transaction (or make the idempotency key itself a
   unique constraint on the effect table) so a crash between the two
   never leaves the key "claimed" without the effect having happened, or
   vice versa.

## What "idempotent processing" means for different operation types

- **Pure creates** (e.g., `POST /reservations`): naturally non-idempotent
  without a key, since calling it twice creates two reservations — this is
  exactly the case that needs a client-supplied idempotency key.
- **Updates by absolute value** (`SET status = 'confirmed'`): naturally
  idempotent already — applying it twice has the same end state as once.
  No key needed.
- **Updates by relative delta** (`UPDATE inventory SET rooms_available =
  rooms_available - 1`): naturally *not* idempotent — retrying doubles the
  decrement. Either make it idempotent by checking a unique constraint
  first (e.g., a `room_holds` row keyed by reservation ID, checked before
  decrementing) or require an idempotency key exactly like a create.
- **Webhook/event consumers**: the sender almost always guarantees
  at-least-once, never exactly-once — the consumer must dedupe on the
  event's own unique ID (not the payload contents, which can vary between
  redeliveries of the "same" event if the sender includes a timestamp).

## Gotchas

- A `SELECT ... FOR UPDATE` followed by a separate check-then-decrement in
  application code is a race condition under concurrent retries even
  though it "looks" atomic — do the check and the write in one SQL
  statement (`UPDATE ... WHERE rooms_available > 0`) or one transaction
  with proper isolation, not read-then-write in two round trips.
- Idempotency keys stored only in application memory (a local cache) don't
  survive a process restart or work across multiple replicas — they need
  to live in the same durable, shared store (Postgres, Redis) the
  operation itself uses, or a retry that lands on a different replica
  after a restart won't be deduped.
- A payment provider's webhook retry and your own client's HTTP retry are
  two *independent* at-least-once delivery paths — dedup logic must handle
  both, and testing only one (e.g., only simulating client retries) misses
  duplicate webhook delivery as a distinct failure mode.
- Returning a different HTTP status code on retry than the original
  response (e.g., original request returns `201 Created`, replayed
  duplicate returns `200 OK` instead of also `201`) breaks clients that
  branch on status code — always replay the exact original response.
- For our partitioned Postgres tables (partitioned by `hotel_id`), an
  idempotency-key uniqueness constraint must include `hotel_id` in its
  scope and in the query's WHERE clause, or the constraint check itself
  will scan every partition instead of pruning to one.

## Real-world grounding

Stripe's Idempotency Keys feature (publicly documented at
stripe.com/docs/api/idempotent_requests) is the most widely cited
production implementation of this pattern: clients pass a
`Idempotency-Key` header on `POST` requests, Stripe stores the key with
the resulting response for 24 hours, and a retried request with the same
key returns the original response rather than creating a second charge —
directly informing similar idempotency-key APIs adopted since (e.g.,
booking, subscription-billing, and payments platforms as a category).

## Verification

- [ ] Idempotency keys are generated client-side, one per logical operation
- [ ] The stored record includes the original response, not just a "seen"
      flag, so retries replay identical results
- [ ] Key uniqueness constraint is enforced atomically with the operation
      it protects (unique DB constraint, same transaction)
- [ ] Relative-delta updates (increment/decrement) are protected by a
      uniqueness check, not assumed idempotent by default
- [ ] Webhook/event consumers dedupe on the event's own ID, independent of
      client-side retry handling