Idempotency and Transaction Integrity in Payment Systems

A payment request can arrive any number of times, and the responsibility for leaving exactly one charge behind sits entirely on the server.

When the same request arrives twice

A payment request is sent once but can arrive many times. When the network drops or a call times out, the client cannot tell whether the authorization succeeded, and the only way to find out is to retry. A user tapping the pay button twice in a row produces the same outcome.

If the server accepts both, one order ends up with two authorizations. What makes this especially bad for card payments is that the money moves much later than the response does. Authorization only places a hold, capture fixes the amount, and settlement moves the funds in a batch. The success screen is not accounting completion, so the duplicate surfaces only hours later at settlement.

The problem starts with the HTTP methods themselves. RFC 7231 defines GET, PUT, and DELETE as idempotent methods, because repeated calls converge the server state to the same value. POST and PATCH, which payment creation and cancellation rely on, change the state on every call and are therefore not idempotent.

Idempotency is the property that reapplying an operation any number of times after the first leaves the result unchanged. It has to be distinguished from safety. A safe method does not modify the resource at all and is therefore always idempotent, but PUT and DELETE are idempotent while still modifying the resource. Giving POST idempotency is not something the protocol does; the server has to implement it.

The idempotency key that absorbs retries

An idempotency key is a unique identifier the client attaches to a request. When the server receives a request with a key it has already seen, it skips the actual processing and returns the response from the first request. The key should be a random value with a low collision probability, such as a UUID v4, and the client must keep the same key across retries.

The key can live in the body, a query parameter, or a header, but the IETF proposes the request header as the standard. Since Stripe introduced the Idempotency-Key header on all POST requests, several payment providers including Toss Payments have adopted the same approach. Placing it in the header lets the front layer stop the request before it reaches domain logic.

sequenceDiagram
    participant C as Client
    participant A as Payment API
    participant K as Key store
    participant P as Authorization
    C->>A: POST /payments (Idempotency-Key k1)
    A->>K: look up k1
    K-->>A: no record
    A->>P: authorization request
    P-->>A: authorization result
    A->>K: store k1 + request fingerprint + response
    A-->>C: 200 OK
    Note over C,A: response is lost and the client times out
    C->>A: retry the same request (Idempotency-Key k1)
    A->>K: look up k1
    K-->>A: stored response
    A-->>C: 200 OK (no re-authorization)

Server behavior splits three ways depending on what the store returns.

Storing only successful responses leaves failed requests unprotected against repeated processing. If a declined authorization is never stored, the retry is handled as a new request and the second attempt can be approved. A failure is also a settled outcome, so it has to be stored alongside successes.

The scope of a key needs a definition too. Toss Payments does not look at the key alone. Sameness is decided from four values together: the idempotency key, the API key, the API address, and the HTTP method (Toss Payments Developer Center, 2023-01). If any one of the four differs, the same key still produces a new request. Without that scope, another merchant that happens to send the same key would receive someone else's response.

The retention period is effectively the key's expiry. Keeping stored responses for around 24 hours and pruning them afterward is the common arrangement. Once the period passes, the same key is processed as a new request, so the client's retry window has to be shorter than the retention period.

Concurrent entry that the key alone cannot stop

There is a gap between looking the key up and storing the result. When two requests carrying the same key arrive at nearly the same moment, both read "no record" and both proceed to authorization. The KakaoPay engineering blog points out that an application cache lookup alone cannot close that instant.

The countermeasure is stacked in layers.

• Distributed lock: a mutex on the idempotency key via Redis SETNX or Redisson lets only the request that claims it enter the authorization path.
• Database unique constraint: a unique constraint on the idempotency key or per-order payment intent permits only one local payment row.
• Processor idempotency key: sending the same key to the external processor makes a repeated call return the existing authorization result.
• Optimistic lock: a version condition on the PENDING → CHARGING → CHARGED state transition blocks a double transition.

Each mechanism protects a different boundary. The distributed lock reduces unnecessary concurrent calls, and the database constraint blocks duplicate local rows. The processor's contract to return the same result for the same key blocks duplicate external authorization.

If the application dies after authorization but before storing the database state, a database transaction cannot recover the external result. A retry must send the same key, receive the earlier authorization result, and converge the local state. A processor without this contract requires authorization lookup, reconciliation, and compensating voids.

Concurrent entry and request mismatch also need to be distinguished in the response code. The IETF specification proposes three.

A 409 means retry shortly, and a 422 means the client reused the key incorrectly. Collapsing both into one code mixes the case that may be retried with the case that requires a code fix.

This logic is better not written per API. Put key lookup, locking, and response storage into a single component shared by payment, cancellation, and refund. Then the domain logic never has to know the key exists.

Saga in the space 2PC leaves behind

A single payment touches several services: order, authorization, inventory, settlement. The classical way to bind them into one atomic transaction is two-phase commit (2PC). A coordinator asks every participant to prepare, and only when all agree does it order the commit.

The problem is that each participant holds locks while waiting for that commit instruction. One slow service ties up the rest, and a dead coordinator stops everyone. 2PC is ruled out in microservice environments because it creates tight coupling and a single point of failure at the same time.

Saga takes the opposite direction. It splits one global transaction into a chain of per-service local transactions, and each step commits immediately. When a later step fails, compensating transactions run in reverse order to undo the earlier commits.

flowchart LR
    subgraph FWD["Forward path"]
        O["Create order<br/>local commit"] --> P["Authorize payment<br/>local commit"] --> S["Deduct inventory<br/>local commit"]
    end
    subgraph COMP["Compensating path"]
        C1["Void authorization"] --> C2["Cancel order"]
    end
    S -->|failure| C1
    P -->|success| V["Post-pivot step<br/>converges by idempotent retry"]

Steps fall into four kinds by character.

The pivot changes the design. Once funds have left for an external institution, a cancellation no longer works. The only way back is a transaction in the opposite direction, and that opposite transaction can fail too. Steps before the pivot must be designed to be compensable, and steps after it must be designed to eventually succeed through retries.

Coordination splits two ways. Orchestration has a central coordinator instruct the next step, while choreography has each service subscribe to events and act on its own. A domain such as payments, where compensating paths are complex, favors orchestration. Which step is current and what needs compensating have to sit in one place before an incident can be traced.

The dual-write trap and Outbox

Each Saga step continues by handing an event to the next one. The point where this breaks is the dual write. A database commit and a broker publish are separate systems and cannot be bound into one transaction.

If the publish fails right after the authorization state is committed, the state remains and the event disappears. The next step never arrives, and compensation never runs either. Reversing the order and publishing first only produces a mismatch in the opposite direction when the commit fails.

The transactional outbox pattern puts the state change and event record in one database transaction. The business state change and the event to be published are inserted into an outbox table within the same transaction. The database then guarantees their atomicity. Publishing is then handled asynchronously in commit order by a change data capture (CDC) connector such as Debezium reading the binlog.

flowchart LR
    subgraph TX["Single database transaction"]
        ST["Payment state change"]
        OB["Outbox record INSERT"]
    end
    TX --> CDC["CDC connector<br/>binlog subscription"]
    CDC --> BR["Event broker"]
    BR --> S1["Settlement service<br/>idempotent consumer"]
    BR --> S2["Reconciliation batch<br/>idempotent consumer"]

The delivery guarantee of this structure is at-least-once. A CDC connector can reread records it has already sent while restarting. That is why idempotency on the consumer side is a premise rather than an option. The consumer has to treat the payment ID or the event ID carried in the event as an idempotency key to keep duplicates out of its store.

The mismatch left at the end and reconciliation

Even with idempotency keys and an outbox in place, mismatches that originate outside the merchant database remain. The most dangerous form in Korean practice is the network cancellation case. The financial institution authorized normally, but a network fault or thread pool exhaustion while the response packet was returning causes a timeout on the merchant side.

At that moment the merchant database says failure while the customer's card says withdrawn. The response is to record the timed-out transaction in an exception state. A retry queue then calls the authorization-void API immediately, forcing both sides back into agreement. The Toss engineering blog adds that it measured the maximum processing rate of the internal authorization server. The timeout boundaries of the front gateway and the integration server were then aligned to that value.

Reconciliation checks for the remaining inconsistencies. This batch process periodically compares the internal ledger against statements from the PSP, the bank, and the card network. It finds omissions, duplicates, and amount mismatches. Exactly-once delivery is not practically achievable, so at-least-once processing, idempotency, and reconciliation are combined to converge on correctness after the fact.

For reconciliation to do its job, the ledger design has to support it. In double-entry bookkeeping every transaction is recorded on a debit and a credit account for the same amount, and the sum is always zero. Instead of updating the balance directly, the ledger accumulates immutable journal entries and derives the balance from them. The value at any point in time can then be recomputed and compared.

The interval is a design choice as well. The zero-downtime ledger migration Toss Payments described ran a correctness verification batch at roughly five-minute intervals. That batch reloaded data that replication lag could have dropped, using the legacy ledger as the source of truth, and resolved the mismatches. With daily reconciliation alone, customers find the mismatches in between before the team does.

Summary

Integrity in a payment system depends on controls at several boundaries. The client and payment processor receive the same idempotency key, while a distributed lock and database unique constraint reduce concurrent entry and local duplication. Saga compensating transactions unwind a chain that failed partway.

Events between services are recorded with local state changes through a transactional outbox and CDC. Consumer idempotency handles the resulting duplicates. Network cancellation handling and periodic reconciliation find and converge any external mismatch that remains.