A customer taps “pay” and the app hangs. They tap it again. Somewhere between the client and your server, one of those two requests got through, or both did, or the first one succeeded but the response never made it back. Multiply that by a flaky mobile connection, a load balancer that times out at 30 seconds, or a background job that retries on any non-200 response, and you have a system that will eventually charge someone twice, create two bookings for the same slot, or send the same command to a device twice. This is not a hypothetical edge case. It is what happens to every API that does not plan for retries, sooner or later.
Idempotency keys are the standard fix, and most engineers know the term. Fewer teams implement them correctly, because the interesting part is not the header field, it is the decisions around it: what counts as “the same request”, how long you remember it, and what you do when the second request looks almost, but not quite, identical to the first.
Why retries are unavoidable, not a bug to fix
It is tempting to treat duplicate requests as a client-side problem: fix the app, debounce the button, stop double-tapping. That helps, but it does not solve the underlying issue. A request can be sent exactly once and still arrive twice, because the network in between does not guarantee delivery or a single round trip. A client can send a request, lose the response to a dropped connection, and correctly conclude it needs to retry, with no way of knowing whether the original request succeeded. Any system with a mobile client, a webhook consumer, a payment gateway, or a queue with at-least-once delivery has this problem built in. The question is not whether duplicates happen, but what your API does when one arrives.
What an idempotency key actually does
The client generates a unique value, typically a UUID, and attaches it to a request that changes state: creating a payment, a booking, an order, a device command. The server treats that value as the identity of the operation, not of the HTTP request. If a request with the same key arrives again, the server does not repeat the side effect. It returns the outcome of the first attempt.
POST /payments HTTP/1.1
Idempotency-Key: 7f3e1c2a-9b4d-4e2a-8c1a-5d6f9e0b1a2c
Content-Type: application/json
{"amount": 4200, "currency": "GBP", "customer_id": "cus_881"}On the server, the sequence that matters is: look up the key, and if it exists, return the stored result without touching the underlying resource again; if it does not exist, record the key and the request fingerprint in the same transaction as the operation itself, then perform the operation. That last part, doing the bookkeeping and the state change atomically, is where most naive implementations fail.
Where teams get it wrong
The failure modes are consistent enough across projects that they are worth listing individually rather than as a general warning.
- Storing the key but not the response. A common half-implementation records that a key was “seen” as a boolean flag, then rejects the retry with a 409 or a generic error. The client now has no way to find out what actually happened to the first request, which is worse than no idempotency handling at all, because it turns a safe retry into a dead end that needs a human to sort out.
- Writing the key and the state change as two separate steps. If the server marks a key as processed and then performs the operation, a crash or a timeout between those two steps leaves the key marked as done while the operation never happened, or the other way round. The key record and the operation need to commit together, inside the same database transaction, or through an outbox pattern if the operation spans systems.
- Scoping the key too broadly or too narrowly. A key that is unique per customer but not per endpoint can accidentally collide a payment attempt with an unrelated refund. A key scoped globally with no expiry eventually becomes a storage problem and makes legitimate repeat operations, a customer paying the same amount again next month, impossible to distinguish from a stuck retry.
- Not validating that the retried request matches the original. If the same key arrives with a different amount or a different payload, returning the cached result from the first request silently processes the wrong operation. The server should compare a fingerprint of the request body against what it stored and reject a mismatch explicitly, rather than assume the client meant to repeat itself.
- Letting the client generate a new key on every retry. This one is a client-side mistake, but it defeats the entire mechanism. If the retry logic creates a fresh UUID each time it resends a request, the server sees two unrelated requests and processes both. The key has to be generated once, before the first attempt, and reused for every retry of that same logical operation.
How long to remember a key
Keys cannot be kept forever, but the retention window needs to outlast any realistic retry. A mobile client that loses connectivity mid-checkout might not retry for minutes, and a background job with exponential backoff might still be trying an hour later. A window that is too short brings back the exact problem idempotency was meant to solve, just delayed. Twenty-four hours is a reasonable default for most interactive operations; asynchronous or offline-capable clients, which is relevant for anything built to work without a constant connection, may need longer. Whatever the window, expiry should be an explicit, documented decision, not whatever the cache eviction policy happens to do.
It is not only about payments
Payment APIs get the most attention because the cost of a duplicate is visible and immediate, but the same pattern applies anywhere a client-initiated action changes state and the network cannot guarantee a clean round trip. A booking or rota system needs it so that a slow response on a shift swap does not create two conflicting entries. A command sent to a fleet of IoT devices needs it so that a retried instruction, issued because an acknowledgement was lost, does not execute twice on the device. A webhook receiver needs the same idea in reverse: the sender may deliver the same event more than once, so the receiver has to deduplicate on its end using whatever identifier the sender provides. The mechanism is the same three questions in every case: what identifies this operation, where is that identity checked, and does the check happen atomically with the effect it is meant to prevent.
What to check before you rely on it
If idempotency handling already exists in a system you are commissioning, extending, or inheriting, these questions separate a working implementation from one that only looks like it works under normal conditions:
- Does a retried request return the original response, or just an error saying it was already processed?
- Is the idempotency key stored and the state change committed in the same transaction, or could a crash between the two leave them inconsistent?
- Is a mismatched retry, same key, different payload, rejected explicitly, or silently accepted?
- What is the retention window for keys, and was it chosen deliberately against realistic client retry behaviour, or left at a framework default?
- Does the client generate the key once per logical operation and reuse it across retries, rather than minting a new one each attempt?
- For inbound webhooks and queue consumers, is there deduplication on the receiving end, not just an assumption that the sender delivers exactly once?
None of this is exotic engineering. It is a handful of decisions made correctly and consistently, which is exactly the kind of detail that is easy to skip under deadline pressure and expensive to discover missing after a customer has been charged twice.