Designing Retry-Safe APIs with Idempotency

Network software must expect duplicate delivery. A mobile app can lose the response after the server completes a purchase; a browser can retry after a timeout; a reverse proxy can retry a request; a user can tap twice; a webhook provider can redeliver by design. For operations such as POST /purchase, POST /reward, POST /create-order, and POST /delete-account, a retry must not silently create a second side effect.

Idempotency is a contract

An operation is idempotent when applying the same intended operation more than once has the same observable outcome as applying it once. HTTP defines GET, PUT, and DELETE as idempotent methods in principle, while POST is normally not. Method choice alone is not enough: a DELETE endpoint that sends a new notification every time still has a duplicate-effect problem.

The Idempotency-Key header lets a server define a retry contract for a POST or PATCH. The client generates a new opaque key for a new intent and preserves that exact key when it retries. MDN's Idempotency-Key reference recommends documenting the endpoints, key requirements, and expiry window.

client -- Idempotency-Key: abc123 --> API
                                      |-- known completed key? --> stored response
                                      |-- otherwise --> transaction --> operation + result

Start with the business invariant

First decide what “once” means. For an order, it might mean one order for a checkout attempt. For a reward, it might mean one grant per user and campaign. For account deletion, it means reaching the same deleted outcome, not repeating every cleanup call. A random UUID has no value if it is not scoped to a clear operation and authenticated principal.

Use a uniqueness constraint as the final line of defense. A conceptual PostgreSQL table could be:

create table idempotency_operations (
  actor_id uuid not null,
  key text not null,
  request_hash text not null,
  status text not null,
  response_json jsonb,
  created_at timestamptz not null default now(),
  primary key (actor_id, key)
);

The actor belongs in the key. Otherwise one user's guessed or reused value could retrieve another user's result. A request hash prevents a different payload from being smuggled under an already-used key: return a conflict rather than replaying a response for a different intent.

A transaction handles the common race

This TypeScript-like example is deliberately simplified. Production code also needs authentication, validation, authorization, response schema handling, and database-specific error mapping.

async function createOrder(actorId: string, key: string, input: CreateOrderInput) {
  const hash = sha256(JSON.stringify(input));
  return database.transaction(async (tx) => {
    const prior = await tx.idempotency.find(actorId, key);
    if (prior?.status === "completed") {
      if (prior.requestHash !== hash) throw new ConflictError("Key reused with another payload");
      return prior.responseJson;
    }

    await tx.idempotency.insert({ actorId, key, requestHash: hash, status: "processing" });
    const order = await tx.orders.insert({ actorId, items: input.items, state: "created" });
    const response = { id: order.id, state: order.state };
    await tx.idempotency.complete(actorId, key, response);
    return response;
  });
}

Two requests can arrive concurrently. A read-then-write check outside a transaction can let both through. The unique key and transaction/locking behavior should make one request the winner; the other re-reads the stored operation or waits according to the chosen policy. On a distributed edge runtime, module memory is not a lock. Use a database constraint/transaction or a coordination primitive such as a Durable Object only when its serialization model fits the workload. Cloudflare documents that Durable Objects provide single-threaded coordination for a given object, but they are not a substitute for a durable business invariant in your data store.

Decide how to represent in-progress work

Returning a stored completed result is easy. An in-progress duplicate needs an explicit contract: wait briefly, return 409 Conflict/202 Accepted with a status URL, or let a job system complete the work. Do not return a fabricated success merely because a row exists.

Choose a retry window and retention policy. Keeping keys forever wastes storage and complicates privacy work; expiring them too quickly permits a late retry to repeat an operation. The right TTL follows the product's retry behavior, client offline duration, payment/webhook needs, and the operation's risk. The uniqueness invariant for a truly irreversible operation may need to outlive the idempotency response cache.

External effects are the hard boundary

A database transaction cannot atomically include a payment provider, email service, or another API. If the database commits and the provider call fails, or the provider succeeds just before your timeout, retries need reconciliation. The transactional outbox pattern records an event with the state change, then a worker delivers it with its own idempotency/deduplication key. Consumers must also deduplicate: webhook delivery is normally at-least-once.

“Exactly once” is usually a misleading shorthand. You can achieve exactly-once effect relative to an invariant inside a database transaction. Across networks and third parties, aim for at-least-once delivery plus idempotent handlers, durable state, and reconciliation. That is more precise and more useful than promising an impossible global guarantee.

Client behavior matters too

Create the ID before the first attempt and persist it with the pending operation if the app can be killed. Do not generate a new key after a timeout; that changes a retry into a new command.

const operation = { id: crypto.randomUUID(), body: { items } };
await pendingOperations.save(operation);

await fetch("/orders", {
  method: "POST",
  headers: { "Content-Type": "application/json", "Idempotency-Key": operation.id },
  body: JSON.stringify(operation.body),
});

Validate and rate-limit before expensive work, but avoid creating a loophole where a duplicate request is charged against a limit differently from its original operation. The secure edge concerns—untrusted input, server authority, and durable rate-limit state—are covered in Building a Secure API with Cloudflare Workers, Zod, and Rate Limiting.

Test ambiguity, not only errors

Response semantics are part of the API

Document what the first and duplicate requests receive. Many APIs replay the original success status and body for a completed matching key. Others return a status resource when work is asynchronous. Both can be sound; surprise is the problem. Also document whether validation failures consume a key. Reserving a key before validation can make a client unable to correct a typo; validating first can allow expensive validation to be repeated. The right answer depends on the endpoint and abuse model.

Keys should be unpredictable opaque values, bounded in length, and treated as request metadata rather than authentication. Do not use a payment reference, email address, or sequential counter as a key. Validate header format early and bind every stored record to the authenticated actor and endpoint. An idempotency key prevents accidental duplicate work; it does not authorize a request, validate inventory, or provide rate limiting.

For queue consumers, carry the original operation identity into each message and make the consumer's write enforce the invariant as well. A producer can be perfectly idempotent while a redelivered message still grants a reward twice downstream. This is why a design review needs to trace the side effect through the API, transaction, outbox, worker, and any provider callback rather than stopping at one HTTP handler.

Test duplicate concurrent requests, a timeout after server commit, process restart between stages, key reuse with changed input, expired keys, and webhook redelivery. Confirm that a retry returns the original result and that an unauthorized caller cannot probe another actor's operation. Idempotency keys are not magic; they are one part of an explicit, durable contract for the unreliable delivery systems every API already lives in.