From 2PC to the Commit Log

The place where consistency is settled has moved from the commit point of 2PC to the commit log. The design question moves with it, from how to coordinate to what must be written down for recovery to work.

Keeping two stores in agreement was long the job of a coordination protocol. Two-Phase Commit (2PC) preserves atomicity by settling the commit only after every participant has prepared. The techniques in use today mostly leave that slot empty. They write a trace into their own commit logs and reconcile afterward.

The same shape appears in unrelated branches of the field. The transactional outbox and Change Data Capture (CDC) moved event publication into the log. Epoch-based optimistic concurrency control pushed coordination out to a batch boundary. Snapshot tables freeze a past fact into stored rows instead of reconstructing it from a query.

There is a price. Coordination is gone, so recovery leans on the log, and recovery that reads only the source side has a limit that has now been proved. This article therefore does not argue for moving to logs. It argues about what has to go into the log afterward for recovery to close.

What filled the gap left by coordination

2PC lost ground in practice because of its failure mode rather than its cost. If the coordinator dies between prepare and commit, the participants wait for a decision while holding locks. That window has no timeout of its own, so the resources stay pinned until the coordinator returns.

Transactions crossing service boundaries were replaced by Saga plus an outbox. Each step becomes an independent transaction and failure is undone by compensation. I compared those options in Idempotency and Transaction Integrity in Payment Systems. What matters here is the next question. What moved into the vacated slot.

Four of these rows share one property. The point where consistency is settled has left the request path, and a log attests to the result. The last column shows the other half of the story. Coordination did not disappear. It changed address.

Four jobs the log absorbed

The outbox moves publication into the log. Instead of sending the event straight to the broker, the application writes it to an event table in the same database, and a separate relay reads that table and publishes it. Committing the business row change and the event row insert in one local transaction makes the database commit the moment the event becomes final. The basic mechanics of the outbox are covered in the earlier article, so I will not repeat them.

CDC goes further and removes even the application's need to write event rows. A tool such as Debezium reads the MySQL binlog or the PostgreSQL WAL directly and produces a change stream. Woowa Brothers commits order domain data and its events in one transaction, then streams them to Kafka through CDC. Because a connector runs only a single task, they split the outbox table per topic to get throughput (Woowa Brothers tech blog, checked 2026-09).

Epoch-based OCC defers coordination itself to a batch boundary. Each region commits locally first and exchanges write-sets per epoch, then deterministically re-executes only the conflicts that surface late. That takes the wide-area round trip out of the commit path. I wrote about how the re-execution cost shifts with the conflict rate in Epoch-Based Optimistic Concurrency Control.

The fourth is backfill. DBLog, published in 2020, copies a whole live database without stopping it by reading chunks in primary-key order and bracketing each chunk with watermarks in the source log. The formal study arXiv:2605.31475 from May 2026 names the state that this replay produces a virtual cut. Replaying a finite prefix yields the same per-key state as the source at a chosen frontier, without ever taking a physical snapshot. The paper states plainly that this establishes neither exactly-once delivery nor destination convergence.

Recovery left open by the source log alone

The limit shows up here. An outbox and CDC move the application's dual write into a relay process, but delivering to the sink and writing the source checkpoint remain separate durable operations. Suppose the process dies right after the remote sink accepts a request and before the checkpoint lands. The restarted recovery cannot tell from local state whether the previous send succeeded.

The Isabelle/HOL formal study arXiv:2608.00501, released in August 2026, supplies a proof for that intuition. Its main result is an information bound. You can construct two reachable post-crash states whose durable source-side state is identical and whose sink acceptance records differ. A recovery policy that observes only the source cannot separate them, so it must choose the same batch in both. That batch either duplicates an effect in one state or leaves one undelivered in the other.

The paper reports that the same conclusion holds for a deterministic deliver-then-checkpoint protocol whose only nondeterminism is crash timing. No amount of careful local schema design or higher isolation gets past this wall. An audit tool can verify the source database perfectly and still not know what the remote receiver accepted, because that fact was never written on the source side.

What the log must carry

The same paper gives the condition that lifts the bound. If the sink acceptance record is authoritative, complete, and current, and if source coordinates distinguish the operations, recovery can compute what is missing. It is a set difference: the obligations demanded by the source log minus what the sink already accepted.

That condition translates into three demands on log design. First, every event needs a strictly monotonic coordinate. It must be a sortable unique identifier such as an LSN or a UUIDv7, so that no payload enters a recovery batch twice. Second, the sink has to expose its acceptance record in a queryable form. A Kafka offset or a consumer-side inbox table plays that part.

Third is the in-flight message. Killing a process does not remove the older messages sitting in socket buffers or retry queues. If one of them lands right after recovery has queried the sink and sent its difference, the sink accepts a duplicate. The paper blocks this with a fence keyed on a generation number. Recovery raises the fence to the next generation as it applies the batch, and the sink rejects anything from a generation below its fence.

The same idea covers concurrent recoverers. When an orchestrator misjudges a live worker as dead and starts a second recoverer, both can follow a correct set-difference policy and still double-fire. The discipline is that a recoverer registers its generation at the sink atomically before it starts, and the sink accepts the write only while that claim still holds. This is the old lesson about locks without fencing tokens, in a new setting.

The lifetime of evidence

Even with all of that in place, the guarantee expires. The last thing the paper covers is how bounded deduplication state and truncated source history limit the lifetime of the guarantee. Real idempotency stores and real log retention windows are finite.

The Stripe API reference states that idempotency keys can be removed automatically once they are at least 24 hours old (checked 2026-09). Reusing a key after the original was pruned generates a new request. An outage that outlasts that window turns a retry into a fresh request. Deduplication does not fail loudly. It quietly expires.

The source side has the same problem. Truncating the commit log destroys the basis for recovery unless the removed range is known to be fully reflected at the sink. Retention is therefore a correctness policy rather than a storage policy. If the deduplication table retains less than the recovery time objective, that difference is the span where nothing is guaranteed.

What does not move into the log

Not every consistency problem rearranges along this axis. There are three counterexamples.

First, anomalies inside a single database are not log problems. Write skew happens when two transactions read different rows and write different rows while their combination breaks an invariant, so snapshot isolation lets it through. The fix is a serializable isolation level or an explicit lock, not log design. The terrain of isolation levels and MVCC is in Database Transactions and MVCC. Transaction propagation inside an application boundary is in Spring Transactions and Propagation.

Second, the information bound stays in force when the sink will not hand over its acceptance record. An external payment gateway takes an idempotency key and blocks a duplicate authorization, but it does not expose a complete ledger our recovery can query. What remains is convergence after the fact, and a reconciliation batch against the external statement fills that role. It does not close recovery. It finds the discrepancy late.

Third, one branch solves convergence through data types rather than logs. Conflict-free Replicated Data Types (CRDTs) rest on algebraic structures with a specific property. Replicas that edit independently reach the same state once they have seen the same set of changes. There is no recovery batch to compute at all, and the cost is a restricted set of expressible operations. I covered the structure in CRDT.

One more qualification: 2PC is not simply wrong. The conditions change when every participant sits under one organization's control and transactions are short. If the coordinator can also be made highly available, 2PC can cost less than operating an outbox, fences, and reconciliation together. That judgment is mine, not the paper's.

Decision criteria

Before deciding to move to a log, there is one question to ask. It is whether recovery closes at this sink.

The third row matters most. At a sink with no acceptance record, every sophisticated design still reduces to choosing between duplication and loss, so decide which cost is larger and record that decision. In the other rows, writing down the right things makes recovery computable.

Operational metrics follow the same split. Duplication and loss have different causes and different responses, so they must not collapse into one consistency metric. Tracking the lag between the source log frontier and the sink offset is the baseline for detecting loss. Duplication surfaces as conflict counts on the sink-side deduplication table.

Summary

The place where distributed consistency is settled has moved from synchronous coordination inside the commit path to the commit log. The outbox and CDC hand publication to the log, epoch-based OCC hands over coordination, and snapshot tables hand over point-in-time preservation. Formal verification has now shown that recovery reading only the source log cannot avoid either duplication or loss. That wall falls only when the sink acceptance record can be queried.

The first thing to check in a design is therefore not the log schema. It is whether the sink surrenders its acceptance record, and whether that record and the idempotency keys outlive the recovery time objective. Where neither holds, decide first which of duplication and loss you are willing to absorb.