Idempotency and Transaction Integrity in Payment Systems

A payment request can arrive any number of times, and the responsibility for leaving exactly one charge behind sits entirely on the server.

When the same request arrives twice

A payment request is sent once but can arrive many times. When the network drops or a call times out, the client cannot tell whether the authorization succeeded, and the only way to find out is to retry. A user tapping the pay button twice in a row produces the same outcome.

If the server accepts both, one order ends up with two authorizations. What makes this especially bad for card payments is that the money moves much later than the response does. Authorization only places a hold, capture fixes the amount, and settlement moves the funds in a batch. The success screen is not accounting completion, so the duplicate surfaces only hours later at settlement.

The problem starts with the HTTP methods themselves. RFC 7231 defines GET, PUT, and DELETE as idempotent methods, because repeated calls converge the server state to the same value. POST and PATCH, which payment creation and cancellation rely on, change the state on every call and are therefore not idempotent.

Idempotency is the property that reapplying an operation any number of times after the first leaves the result unchanged. It has to be distinguished from safety. A safe method does not modify the resource at all and is therefore always idempotent, but PUT and DELETE are idempotent while still modifying the resource. Giving POST idempotency is not something the protocol does; the server has to implement it.

The idempotency key that absorbs retries

An idempotency key is a unique identifier the client attaches to a request. When the server receives a request with a key it has already seen, it skips the actual processing and returns the response from the first request. The key should be a random value with a low collision probability, such as a UUID v4, and the client must keep the same key across retries.

The key can live in the body, a query parameter, or a header, but the IETF proposes the request header as the standard. Since Stripe introduced the Idempotency-Key header on all POST requests, several payment providers including Toss Payments have adopted the same approach. Placing it in the header lets the front layer stop the request before it reaches domain logic.

Server behavior splits three ways depending on what the store returns.

Storing only successful responses gives half a defense. If a declined authorization is never stored, the retry is handled as a new request and the second attempt can be approved. A failure is also a settled outcome, so it has to be stored alongside successes.

The scope of a key needs a definition too. Toss Payments does not look at the key alone. Sameness is decided from four values together: the idempotency key, the API key, the API address, and the HTTP method (Toss Payments Developer Center, 2023-01). If any one of the four differs, the same key still produces a new request. Without that scope, another merchant that happens to send the same key would receive someone else's response.

The retention period is effectively the key's expiry. Keeping stored responses for around 24 hours and pruning them afterward is the common arrangement. Once the period passes, the same key is processed as a new request, so the client's retry window has to be shorter than the retention period.

Concurrent entry that the key alone cannot stop

There is a gap between looking the key up and storing the result. When two requests carrying the same key arrive at nearly the same moment, both read "no record" and both proceed to authorization. The KakaoPay engineering blog points out that an application cache lookup alone cannot close that instant.

The countermeasure is stacked in layers.

• Distributed lock: a mutex on the idempotency key via Redis SETNX or Redisson, so only the request that claims it proceeds.
• Database unique constraint: a constraint such as UNIQUE(order_id, charge_status) rolls back the transaction that arrives late on a constraint violation.
• Optimistic lock: adding a version condition to the PENDING → CHARGING → CHARGED state machine UPDATE blocks a double transition.

The three play different roles. The lock is a performance device that strips unnecessary authorization calls off the front, while the final guarantee of correctness is the database unique constraint. Even if Redis dies, the constraint remains inside the database and keeps working.

Concurrent entry and request mismatch also need to be distinguished in the response code. The IETF specification proposes three.

A 409 means retry shortly, and a 422 means the client reused the key incorrectly. Collapsing both into one code mixes the case that may be retried with the case that requires a code fix.

This logic is better not written per API. Put the key lookup, locking, and response storage into a single component shared by payment, cancellation, and refund, and the domain logic never has to know the key exists.

Saga in the space 2PC leaves behind

A single payment touches several services: order, authorization, inventory, settlement. The classical way to bind them into one atomic transaction is two-phase commit (2PC). A coordinator asks every participant to prepare, and only when all agree does it order the commit.

The problem is that each participant holds locks while waiting for that commit instruction. One slow service ties up the rest, and a dead coordinator stops everyone. 2PC is ruled out in microservice environments because it creates tight coupling and a single point of failure at the same time.

Saga takes the opposite direction. It splits one global transaction into a chain of per-service local transactions, and each step commits immediately. When a later step fails, compensating transactions run in reverse order to undo the earlier commits.

Steps fall into four kinds by character.

The pivot changes the design. Once funds have left for an external institution, a cancellation no longer works. The only way back is a transaction in the opposite direction, and that opposite transaction can fail too. Steps before the pivot must be designed to be compensable, and steps after it must be designed to eventually succeed through retries.

Coordination splits two ways. Orchestration has a central coordinator instruct the next step, while choreography has each service subscribe to events and act on its own. A domain such as payments, where compensating paths are complex, favors orchestration. Which step is current and what needs compensating have to sit in one place before an incident can be traced.

The dual-write trap and Outbox

Each Saga step continues by handing an event to the next one. The point where this breaks is the dual write. A database commit and a broker publish are separate systems and cannot be bound into one transaction.

If the publish fails right after the authorization state is committed, the state remains and the event disappears. The next step never arrives, and compensation never runs either. Reversing the order and publishing first only produces a mismatch in the opposite direction when the commit fails.

The transactional outbox pattern pushes the problem inside the database. When the business state change and the event to be published are inserted into an outbox table within the same transaction, the database guarantees their atomicity. Publishing is then handled asynchronously in commit order by a change data capture (CDC) connector such as Debezium reading the binlog.

The delivery guarantee of this structure is at-least-once. A CDC connector can reread records it has already sent while restarting. That is why idempotency on the consumer side is a premise rather than an option. The consumer has to treat the payment ID or the event ID carried in the event as an idempotency key to keep duplicates out of its store.

The mismatch left at the end and reconciliation

Even with idempotency keys and an outbox in place, mismatches that originate outside our database remain. The most dangerous form in Korean practice is the network cancellation case. The financial institution authorized normally, but a network fault or thread pool exhaustion while the response packet was returning causes a timeout on the merchant side.

At that moment the merchant database says failure while the customer's card says withdrawn. The response is to record the timed-out transaction in an exception state. A retry queue then calls the authorization-void API immediately, forcing both sides back into agreement. The Toss engineering blog adds that it measured the maximum processing rate of the internal authorization server. The timeout boundaries of the front gateway and the integration server were then aligned to that value.

The last net for correctness is reconciliation. It is a batch that periodically compares the internal ledger against statements from the PSP, the bank, and the card network to find omissions, duplicates, and amount mismatches. Exactly-once delivery is not practically achievable, so at-least-once processing, idempotency, and reconciliation are combined to converge on correctness after the fact.

For reconciliation to do its job, the ledger design has to support it. In double-entry bookkeeping every transaction is recorded on a debit and a credit account for the same amount, and the sum is always zero. Instead of updating the balance directly, the ledger accumulates immutable journal entries and derives the balance from them. The value at any point in time can then be recomputed and compared.

The interval is a design choice as well. The zero-downtime ledger migration Toss Payments described ran a correctness verification batch at roughly five-minute intervals. That batch reloaded data that replication lag could have dropped, using the legacy ledger as the source of truth, and resolved the mismatches. With daily reconciliation alone, customers find the mismatches in between before the team does.

Summary

Integrity in a payment system is not built in a single layer. The idempotency key sent by the client absorbs retries, while a distributed lock and a database unique constraint block concurrent entry. Saga compensating transactions unwind a chain that failed partway.

Events between services must be bound to the database transaction through a transactional outbox and CDC to avoid loss. Consumer idempotency catches the duplication that comes as the price. Whatever external mismatch still remains is cleared after the fact by network cancellation handling and periodic reconciliation. In the order the layers stack, the idempotency key comes first, database constraints and event delivery follow, and reconciliation is the last layer.