API Idempotency: Why the Retry Button Charged Them Twice

Published on September 3, 20265 min read

A user taps Pay. The spinner runs. Their connection drops and the request times out. They tap Pay again.

Did the first request reach your server?

Nobody knows. Not you, not the client. A missing response is not evidence of a missing request. Maybe the payment never happened. Maybe it happened perfectly and only the reply got lost.

That gap is where double charges live.

The Third Outcome

Every network call has three outcomes, not two: success, failure, and unknown. The first two tell you what happened on the server. The third tells you nothing, and it happens a lot. Mobile networks drop, proxies time out, containers get rescheduled mid-request.

The client has two options and both are wrong. Retry, and risk doing the work twice. Don't retry, and risk doing it zero times while the user believes it happened.

Idempotency removes the choice.

The Definition

An operation is idempotent when applying it more than once has the same effect as applying it once. Not necessarily the same response, the same effect on your data.

Some Methods Are Already Idempotent

GET, PUT, and DELETE are idempotent by contract. POST is not. PATCH depends on how you wrote it.

The rule behind that: absolute operations are idempotent and relative ones are not.

the-difference.ts
// Idempotent: run it a hundred times, balance is 100
await db.update(accounts).set({ balance: 100 }).where(eq(accounts.id, id));

// Not idempotent: run it a hundred times, you owe someone an apology
await db.update(accounts).set({ balance: sql`balance + 100` }).where(eq(accounts.id, id));

Idempotent Is Not the Same as Unchanged

DELETE /orders/42 twice is idempotent. The order is gone either way. The second call may honestly return 404 while the first returned 204. Different status codes, same end state. Idempotency constrains the data, not the response.

So the problem is narrower than it looked. What you need to solve is POST: charging cards, creating orders, sending invites.

Idempotency Keys

Let the client name the operation. It generates a unique ID before sending, and reuses the same ID on every retry:

request
POST /v1/payments HTTP/1.1
Content-Type: application/json
Idempotency-Key: 8f14e45f-ea56-4a2c-b6e1-9b0c2f3d1a77

{ "amount": 4200, "currency": "usd", "customer": "cus_9fK2" }

The server looks the key up, and there are three cases:

Never seen it: run the operation, store the response, return it.
Seen it and it completed: skip the work and return the stored response.
Seen it and it is still running: return 409 and let the client retry shortly.

The Key Belongs to the Client

The server can't generate it. A retry would get a new one and look like a new request. A UUID v4 is fine. Deriving it from the request body is not: two genuinely separate $42 payments to the same person would collide.

Implementing It

services/payments.ts
export const createPayment = async (
  key: string,
  payload: PaymentPayload
): Promise<Result<Payment>> => {
  // Claim the key first. The unique constraint is what makes this safe:
  // two concurrent requests race, exactly one wins the insert.
  const claimed = await db
    .insert(idempotencyKeys)
    .values({ key, status: 'processing', requestHash: hash(payload) })
    .onConflictDoNothing()
    .returning();

  if (claimed.length === 0) return replayExisting(key, payload);

  try {
    const payment = await chargeCard(payload);

    await db
      .update(idempotencyKeys)
      .set({ status: 'completed', response: payment })
      .where(eq(idempotencyKeys.key, key));

    return { ok: true, data: payment };
  } catch (error) {
    console.error('Error in createPayment::', error);
    // Release the key so a retry can genuinely try again.
    await db.delete(idempotencyKeys).where(eq(idempotencyKeys.key, key));
    return { ok: false, errorKey: 'PAYMENT_FAILED' };
  }
};

Three details carry the weight here:

The insert comes first. Claiming the key before charging is what closes the race. Charge first and record after, and two simultaneous requests both find no key and both charge.

onConflictDoNothing is the lock. Not a mutex, not a Redis SETNX, not check-then-insert. A unique index is the only thing here that is actually atomic under concurrency.

Failures release the key. A card decline shouldn't permanently poison it. A successful charge keeps it.

On replay, compare a hash of the request body before returning the stored response. Without that check, a client bug that reuses a key for a $5 charge silently returns the receipt for a $4,200 one, and the $5 charge never happens.

Store the Response, Not Just the Key

A retry must return what the original returned. If you only record "this key is done" and return an empty 200, the client never learns the payment ID it was waiting for.

Keys expire. 24 hours is standard, and it comfortably outlives any legitimate retry. Without expiry the table grows forever.

The Same Problem, Different Clothes

Webhooks. Stripe and GitHub retry on non-2xx, which guarantees at-least-once delivery. Duplicates are normal, not exceptional. Every payload carries an event ID, so insert it with onConflictDoNothing and skip the ones you have already handled.

Message queues. SQS and Kafka are also at-least-once. A consumer that crashes after doing the work but before acknowledging will see the message again.

Job schedulers. Cron overlaps when a nightly job runs longer than expected.

At-Least-Once Plus Idempotency Equals Exactly-Once

Exactly-once delivery isn't something you can buy, because the network won't allow it. At-least-once delivery on top of idempotent handling produces exactly-once effects, which is the property you actually wanted.

Retrying on the Client

Idempotency makes retries safe. It doesn't make them polite.

Generate the key once, outside the retry loop. Move it inside and you have rebuilt the double-charge bug with extra steps.

Retry only on 408, 429, and 5xx, and back off exponentially with jitter. If ten thousand clients all wait exactly 400ms, they come back as one synchronized wave and knock the service over again.

Where It's Worth It

Not every endpoint. Only the ones where a duplicate costs something:

Money moving.
Records you can't easily un-create.
Anything leaving your system: email, SMS, partner APIs.
Counters.

For a PUT that sets a profile field, HTTP already gave you idempotency.

The Takeaway

A timeout is not a failure, it is an unknown. Clients faced with an unknown will retry, and that is correct behavior. Your job is to make it harmless.

Let the client name the operation. Claim the key with a unique constraint rather than application logic. Store the response so replays return the same answer. Expire the keys.

Do that, and the retry button stops being the scariest thing in your product.

All articles