Idempotency keys make payments, webhooks and retries safe. Learn to design idempotent POST endpoints, claim keys atomically, store responses, and TTL them.
A payment succeeds on Stripe's side, but the response times out before it reaches your server. Your client retries. Now you've either charged the customer twice or you're staring at a support ticket at 2 AM. I've been on the wrong side of this exact scenario in production, and the fix is almost embarrassingly simple: idempotency keys. It's the one habit that turns "retries are dangerous" into "retries are free."
This is a deep dive on how to actually design idempotent endpoints — not the hand-wavy "just make it idempotent" advice — and how to store the keys without provisioning a giant table that outgrows your database. I'll cover the race condition you will hit under load, how idempotency composes across services, the difference between idempotency and plain deduplication, and the TTL trick that keeps your key store self-cleaning instead of turning into a permanent ledger.
Here's the uncomfortable truth about any networked system: you cannot tell the difference between a request that failed and a response that got lost. This is a fundamental property of unreliable networks — the classic "Two Generals" problem in a business suit. When your mobile client fires a POST /charges and the socket dies before the response comes back, the client has no idea whether the server processed the charge or never received it. Its only sane options are to retry or to give up — and giving up on a payment is worse than retrying.
So retries happen whether you design for them or not. They come from:
2xx. A slow database on your side means the same event arrives three times.At-least-once delivery is the default everywhere that matters, and "exactly-once" is mostly a marketing word. The real engineering answer isn't "prevent duplicates" — it's make duplicates harmless. That reframing is the whole point. You stop fighting the network and start designing endpoints that don't care how many times they're called. That's what idempotency buys you.
A quick precision check, because people conflate two things. In HTTP semantics, GET, PUT and DELETE are already idempotent by definition — calling them N times leaves the same server state as calling them once. The problem child is POST, which is supposed to create a new resource on every call. Idempotency keys retrofit POST with the safety of PUT.
It's also worth separating idempotency from simple deduplication. Deduplication just drops the second request. True idempotency is stronger: the second caller gets back the exact same response as the first — same status code, same resource ID, same body — as if it were the only request that ever happened. That distinction matters, because a client that retries usually needs the answer (the charge ID, the created order), not just a silent "already handled."
The contract is: the caller generates a unique key (a UUID v4 is fine) and sends it with the request. The first time the server sees that key, it does the work and records the result against the key. Every subsequent request carrying the same key returns the stored result without re-running the side effect.
POST /charges HTTP/1.1Idempotency-Key: 8f14e45f-ea51-4c3b-9a2d-1f3b7c8d9e00Content-Type: application/json{ "amount": 4999, "currency": "aed", "customer": "cus_123" }The key is per-operation — not per-user, not per-endpoint. It scopes a single logical intent: "charge this customer 49.99 AED, once." A crucial design rule follows from this: the client generates the key before the first attempt and reuses it across every retry of that same intent. If the client mints a fresh UUID on each retry, you've built an elaborate mechanism that guarantees nothing. The key must be stable for the lifetime of the operation.
The naive implementation has a race condition you will absolutely hit under load. If you do SELECT key → if not found → do work → INSERT key, two concurrent retries can both pass the SELECT before either one inserts. Now you've done the work — charged the card — twice. I've watched this happen with duplicate webhook deliveries arriving milliseconds apart, which is exactly when it hurts most.
The fix is to claim the key atomically before doing any work. In SQL, that means a UNIQUE constraint on the key column plus an insert-first pattern (INSERT ... ON CONFLICT DO NOTHING in Postgres): whoever wins the insert owns the operation; the loser reads the stored result. In a Firestore-backed backend — which is most of what I run, since I keep our Firebase bill at zero and avoid Cloud Functions where I can — a transaction gives you the same atomicity through optimistic concurrency:
Future<ChargeResult> charge(String idemKey, ChargeRequest req) async { final keyRef = firestore.collection('idem_keys').doc(idemKey); return firestore.runTransaction((tx) async { final snap = await tx.get(keyRef); if (snap.exists) { final data = snap.data()!; // Guard against key reuse with a *different* payload. if (data['requestHash'] != sha256(req)) { throw UnprocessableEntityException( 'Idempotency-Key reused with a different request body', ); } if (data['status'] == 'completed') { // Replay the stored response — no re-charge. return ChargeResult.fromJson(data['response']); } // A concurrent request already claimed this key. throw ConflictException('Request in progress for $idemKey'); } // Claim the key atomically. If two requests race, the // transaction's optimistic concurrency forces one to retry. tx.set(keyRef, { 'status': 'in_progress', 'requestHash': sha256(req), 'createdAt': FieldValue.serverTimestamp(), }); // The transaction commits the claim; do the side effect after. return _runChargeAndPersist(keyRef, req); });}Three details matter more than they look:
422 Unprocessable Entity, don't silently replay the wrong result. This catches key collisions that would otherwise let one operation's response leak into a completely different request. This is exactly how Stripe's own API behaves.done flag. The whole point is that the retry gets the same answer, including the resource ID you created. A boolean flag forces the client to go re-fetch and guess which record was theirs.in_progress state explicitly. While the first request is still doing work, a retry should get a 409 Conflict, not a green light and not a second charge. The caller backs off and retries, by which point you have a stored result to replay. Without this state, two near-simultaneous requests both see "no completed key" and both proceed.Here's the subtle part idempotency keys don't magically solve: if your side effect calls another system (Stripe, a bank, an email or SMS provider), you can crash after charging the card but before recording completion. Now your own key is stuck at in_progress forever, and a retry gets a permanent 409. You've traded a double charge for a stuck operation.
The clean answer is to push idempotency down to the external call too. Stripe, for example, accepts its own Idempotency-Key header. Reuse the same key you were handed:
final charge = await stripe.charges.create( amount: req.amount, currency: req.currency, customer: req.customer, options: RequestOptions(idempotencyKey: idemKey), // reuse the caller's key);
Now even if you replay the entire flow after a crash, Stripe deduplicates on its end and returns the same charge ID it created the first time. You record that ID and flip your own key to completed. Idempotency composes — each layer only has to be safe against replays of itself, and the whole chain becomes replay-safe. That composition property is the genuinely beautiful part, and it's why I treat "does this API accept an idempotency key?" as a hard requirement when I pick a payment or messaging provider.
For side effects you don't control this way (like a fire-and-forget email), the fallback is to make the recording step and the trigger step converge on a background worker driven off your own durable state, so a crash just means the worker re-runs and the downstream provider's own dedup catches it.
The objection I always hear: "won't this table grow forever and wreck my database?" Yes — if you never delete anything. But idempotency keys have a naturally short useful life. A retry that arrives a week later isn't a retry; it's a new intent or a bug. Keys only need to outlive the retry window of your callers, which is typically minutes to a day.
So the storage strategy is a short-lived cache, not a permanent ledger:
EXPIRE), DynamoDB (TTL attribute) and Postgres (a scheduled DELETE or partition drop) all give you the same thing. I set expiry comfortably longer than the longest retry window I care about — 24 hours is generous for almost everyone. After that, the key silently evaporates. No cron job, no cleanup service to babysit.The key insight: idempotency keys are a time-bounded promise, not permanent state. Once every plausible caller has stopped retrying, the record has done its job. TTL turns "giant table that needs a cleanup job" into "self-cleaning cache" for free.
One caveat worth stating: if you need a permanent record that an operation happened — for audit, accounting, or reconciliation — that belongs in a real ledger table, not in the idempotency store. Keep the two concerns separate. The idempotency key protects the write path from duplicates; the ledger is your durable source of truth.
When I add a new money-moving or webhook-consuming endpoint, I run down this list:
422?in_progress and return 409 while the first call is running?SELECT-then-INSERT pattern has a race that causes double charges. Use a unique constraint or a transaction.in_progress. These three details turn a fragile flag into a correct replay-safe endpoint.If you build anything that moves money, sends webhooks, or lives behind a retrying queue — which is nearly everything — bake idempotency in from the first endpoint. The habit is small: accept a key, claim it atomically before doing work, store the full response, push the key to downstream systems, and TTL the record. Do that and retries stop being a source of 2 AM incidents and become what they should be — a boring, safe reliability tool.