From 2PC to the Commit Log

The place where consistency is settled has moved from the commit point of 2PC to the commit log. The design question moves with it, from how to coordinate to what must be written down for recovery to work.

Keeping two stores in agreement was long the job of a coordination protocol. Two-Phase Commit (2PC) preserves atomicity by settling the commit only after every participant has prepared. The techniques in use today mostly leave that slot empty. They write a trace into their own commit logs and reconcile afterward.

The same shape appears in unrelated branches of the field. The transactional outbox and Change Data Capture (CDC) moved event publication into the log. Epoch-based optimistic concurrency control pushed coordination out to a batch boundary. Snapshot tables freeze a past fact into stored rows instead of reconstructing it from a query.

There is a price. Coordination is gone, so recovery leans on the log, and recovery that reads only the source side has a limit that has now been proved. This article therefore does not argue for moving to logs. It argues about what has to go into the log afterward for recovery to close.

What filled the gap left by coordination

2PC lost ground in practice because of its failure mode rather than its cost. If the coordinator dies between prepare and commit, the participants wait for a decision while holding locks. That window has no timeout of its own, so the resources stay pinned until the coordinator returns.

Transactions crossing service boundaries commonly use Saga plus an outbox. Each step becomes an independent transaction, and compensation reverses a failed step. Idempotency and Transaction Integrity in Payment Systems compares those options.

This article examines what filled the gap left by synchronous coordination.

Four of these rows share one property. Consistency is settled outside the request path, and a log records the result. The last column shows where coordination now occurs.

Four jobs the log absorbed

The outbox moves publication into the log. The application writes the event to an event table in the same database instead of sending it directly to the broker. A separate relay reads that table and publishes the event. Committing the business row change and the event row insert in one local transaction makes the database commit the moment the event becomes final. The basic mechanics of the outbox are covered in the earlier article, so I will not repeat them.

CDC goes further and removes even the application's need to write event rows. A tool such as Debezium reads the MySQL binlog or the PostgreSQL WAL directly and produces a change stream. Woowa Brothers commits order domain data and its events in one transaction, then streams them to Kafka through CDC. Because a connector runs only a single task, they split the outbox table per topic to get throughput (Woowa Brothers tech blog, checked 2026-09).

Epoch-based OCC defers coordination itself to a batch boundary. Each region commits locally first and exchanges write-sets per epoch, then deterministically re-executes only the conflicts that surface late. That takes the wide-area round trip out of the commit path. I wrote about how the re-execution cost shifts with the conflict rate in Epoch-Based Optimistic Concurrency Control.

The fourth is backfill. DBLog, published in 2020, copies a whole live database without stopping it by reading chunks in primary-key order and bracketing each chunk with watermarks in the source log. The formal study arXiv:2605.31475 from May 2026 names the state that this replay produces a virtual cut. Replaying a finite prefix yields the same per-key state as the source at a chosen frontier, without ever taking a physical snapshot. The paper states plainly that this establishes neither exactly-once delivery nor destination convergence.

Recovery left open by the source log alone

The limit shows up here. An outbox and CDC move the application's dual write into a relay process, but delivering to the sink and writing the source checkpoint remain separate durable operations. Suppose the process dies right after the remote sink accepts a request and before the checkpoint lands. The restarted recovery cannot tell from local state whether the previous send succeeded.

The Isabelle/HOL formal study arXiv:2608.00501, released in August 2026, supplies a proof for that intuition. Its main result is an information bound. You can construct two reachable post-crash states whose durable source-side state is identical and whose sink acceptance records differ. A recovery policy that observes only the source cannot separate them, so it must choose the same batch in both. That batch either duplicates an effect in one state or leaves one undelivered in the other.

flowchart LR
  subgraph SRC["Source side (visible to recovery)"]
    H["Commit log<br/>coordinate identifying each operation"]
    C["Relay checkpoint<br/>how far it has sent"]
  end
  W["Wire channel<br/>delayed and reordered in-flight messages"]
  subgraph SNK["Sink side (must be queried)"]
    F["Arrival fence<br/>generation number check"]
    E["Acceptance record<br/>what was actually accepted"]
  end
  H --> C --> W --> F --> E
  E -. "set difference gives the remaining obligations" .-> H

The paper reports that the same conclusion holds for a deterministic deliver-then-checkpoint protocol whose only nondeterminism is crash timing. Careful local schema design and higher isolation cannot overcome this information limit. An audit tool can verify the source database perfectly and still not know what the remote receiver accepted, because that fact was never written on the source side.

What the log must carry

The same paper gives the condition that lifts the bound. If the sink acceptance record is authoritative, complete, and current, and if source coordinates distinguish the operations, recovery can compute what is missing. It is a set difference: the obligations demanded by the source log minus what the sink already accepted.

That condition translates into three demands on log design. First, each operation needs a stable coordinate that distinguishes it. A log sequence number (LSN) identifies order within a log, while UUIDv7 primarily supports sorting by creation time. Strict monotonicity within the same millisecond depends on the implementation (RFC 9562 §6.2).

A unique coordinate does not provide deduplication by itself. Retries must retain the original operation identifier, and the sink must enforce uniqueness on that identifier. Second, the sink has to expose its acceptance record in a queryable form. A Kafka offset or a consumer-side inbox table provides that record.

Third is the in-flight message. Killing a process does not remove older messages from socket buffers or retry queues. If one lands just after recovery queries the sink and sends its difference, the sink accepts a duplicate.

The paper blocks this with a fence keyed on a generation number. Recovery raises the fence as it applies the batch, and the sink rejects messages from an older generation.

The same idea covers concurrent recoverers. Suppose an orchestrator misjudges a live worker as dead and starts a second recoverer. Both can follow a correct set-difference policy and still send duplicates. The discipline is that a recoverer registers its generation at the sink atomically before it starts, and the sink accepts the write only while that claim still holds. This is the old lesson about locks without fencing tokens, in a new setting.

The lifetime of evidence

Even with all of that in place, the guarantee expires. The last thing the paper covers is how bounded deduplication state and truncated source history limit the lifetime of the guarantee. Real idempotency stores and real log retention windows are finite.

The Stripe API reference states that idempotency keys can be removed automatically once they are at least 24 hours old (checked 2026-09). Reusing a key after the original was pruned generates a new request. An outage that outlasts that window turns a retry into a fresh request. Deduplication expires without an explicit failure.

The source side has the same problem. Truncating the commit log destroys the basis for recovery unless the removed range is known to be fully reflected at the sink. Retention is therefore a correctness policy rather than a storage policy. If the deduplication table retains less than the recovery time objective, that difference is the span where nothing is guaranteed.

What does not move into the log

Not every consistency problem rearranges along this axis. There are three counterexamples.

First, anomalies inside a single database are not log problems. Write skew occurs when transactions read and write different rows while their combined result breaks an invariant. Snapshot isolation can permit it, so the fix is serializable isolation or an explicit lock.

Database Transactions and MVCC explains the isolation levels. Spring Transactions and Propagation covers propagation inside an application boundary.

Second, the information bound stays in force when the sink will not hand over its acceptance record. An external payment gateway may accept an idempotency key and block a duplicate authorization without exposing a complete ledger to the recovery process. Reconciliation against the external statement can find the discrepancy later, but it cannot close recovery at delivery time.

Third, one branch solves convergence through data types rather than logs. Conflict-free Replicated Data Types (CRDTs) rest on algebraic structures with a specific property. Replicas that edit independently reach the same state once they have seen the same set of changes. There is no recovery batch to compute at all, and the cost is a restricted set of expressible operations. I covered the structure in CRDT.

2PC also fits some conditions. All participants may sit under one organization's control, transactions may be short, and the coordinator may be highly available. In that case, operating 2PC can cost less than running an outbox, fences, and reconciliation together. This is an operational judgment rather than a conclusion from the paper above.

Decision criteria

Before deciding to move to a log, there is one question to ask. It is whether recovery closes at this sink.

The third row matters most. At a sink with no acceptance record, even a sophisticated design must choose between duplication and loss. Decide which cost is larger and record that decision. In the other rows, writing down the right things makes recovery computable.

Operational metrics follow the same split. Duplication and loss have different causes and different responses, so they must not collapse into one consistency metric. Tracking the lag between the source log frontier and the sink offset is the baseline for detecting loss. Duplication surfaces as conflict counts on the sink-side deduplication table.

Summary

The place where distributed consistency is settled has moved from synchronous coordination inside the commit path to the commit log. The outbox and CDC hand publication to the log, epoch-based OCC hands over coordination, and snapshot tables hand over point-in-time preservation. Formal verification has now shown that recovery reading only the source log cannot avoid either duplication or loss. That wall falls only when the sink acceptance record can be queried.

The first thing to check in a design is therefore not the log schema. It is whether the sink surrenders its acceptance record, and whether that record and the idempotency keys outlive the recovery time objective. Where neither holds, decide first which of duplication and loss you are willing to absorb.