Transactions across leaders

COMMIT ACROSS LEADERS: one transaction writing ranges that different nodes lead, committed whole or not at all — how it commits, what a reader sees, what the caller is told, and what it costs.

When two nodes lead two ranges, each takes its own writes, and a transaction is normally judged on the one node that leads everything it writes. A transaction that writes records on both is refused SpansLeaderships, naming both nodes. From 0.24.0-beta it can ask to be committed across them instead, whole or not at all:

BEGIN;
  UPDATE accounts:'a17' SET balance -= 40;
  UPDATE accounts:'h17' SET balance += 40;
COMMIT ACROSS LEADERS;

The clause goes on the COMMIT, or on a single write — UPDATE … ACROSS LEADERS — beside ACKNOWLEDGE and in either order. A write inside BEGIN that says it lets the transaction's COMMIT reach across leaders as though the COMMIT had said it.

It is permission, never a cost. A transaction that asks and turns out to write one leader's ranges commits in one step, exactly as it would have without the clause. Only one that really spans leaders pays for it.

How it commits

Step 1 is one round: the record and every part go out together, each held by a majority of its range's voters. The caller hears the outcome at 2; step 3 follows behind the answer and waits for nobody.

One round. The node the transaction ran on drives it. In a single round it writes the transaction's record STAGING — naming every range the transaction writes — on the leader of the first range, together with that range's own writes, and sends every other leader its range's writes to prepare. Each leader runs on them every check its own commit runs — its fence, the schema, uniqueness, bounds, the user's grants — and the conflict check against everything committed there since the transaction read, and answers once a majority of its range's voters hold the writes as intents.
Answer. A transaction whose record stages and whose every part is held by a majority is committed — nothing can take that back — so the caller is told committed as soon as the last part is held.
Behind the answer. The record is turned COMMITTED and each leader turns its intents into ordinary versions in its own log. Nobody waits for this step; anyone may repeat it, and doing it twice changes nothing. If it is lost, the outcome is worked out again from the parts.

A refusal from any leader, or an answer that never came, does not abort the transaction outright: the part may have landed after all. The driver asks every leader whether its part landed and bars one that has not, so it can never land later; then the transaction is committed if every part landed, and aborted otherwise.

Every step waits for a majority whatever ACKNOWLEDGE says. A prepare or an outcome that a failover could lose would not be one.

What the caller is told

Committed, or one of:

refusal	what it means	what to do
`AcrossAborted`	a leader refused its part — the refusal is quoted — or the record was aborted before the decision landed. Nothing applies anywhere.	a conflict is retriable as it always is; anything else is the refusal it quotes
`AcrossInDoubt`	an answer never came, and the node could not learn which way the transaction went	it is committed if every part landed and aborted if not, and the record settles it. Read the records to learn which; an idempotent retry (`UPSERT`) is safe, a `CREATE` repeated is refused as already there
`AcrossNotHeldHere`	the node it ran on holds no copy of a range it writes	run it on a node that holds every range it writes — half of the conflict check is made on that copy
`AcrossUnavailable`	the node knows no peers to carry the parts to	run it on a clustered node
`AcrossKind`	it writes a vault, bucket, space, topic, queue, series, or vector or geo store, each of which keeps something beside its records that a prepared write would bypass	keep those writes in one leader's ranges; only tables, collections and edges commit across leaders

Over HTTP an abort answers with the status of the refusal behind it, judged on the node that refused: 409 for a conflict, a lapsed record or a leader not reached — worth retrying — 403 for a grant the caller lacks, and 400 for a write the schema refuses. In doubt is 409, like a commit a majority did not confirm in time: read the records before deciding what to do.

What a reader sees

Never half of it. Each read decides once, from its own snapshot, whether the whole transaction is visible; when it is not, it reads the version under every one of the transaction's records. A transaction is therefore whole in every read of it, or absent from it.

That is atomic visibility, not one order across the cluster. There is no clock shared by every node, so two readers on two ranges are not ordered against each other, and write skew is admitted as it is under the store's snapshot isolation. What is promised is that no reader ever sees one part of a transaction without the rest.

While it is in flight, an intent is a lock: another write to a record holding one is refused Conflict — retriable, and the message says an intent refused it — rather than made to wait. An index over a table holding such a transaction's records answers through a checked path until all its parts have landed on the node, and a graph traversal over its edges is refused AcrossSettling, retriable, for that moment.

When a node dies part-way

The record, the intents and the decision are log records held by a majority, so whoever leads each range next continues from them — a leader change moves nothing that matters, and a new leader answers for its range only once its own first entry is held by a majority. A driver that dies leaves its record STAGING; once the record's liveness has lapsed — four rounds of the failover policy — the record's leader recovers it as the driver would have: every part landed means committed, and a part that has not is barred first and the transaction aborted. Exactly one outcome is ever recorded and no intent is held for ever. A decided record is forgotten once every leader's resolution is itself held by a majority.

A snapshot backup carries such a transaction as its moment saw it, so a restored store answers as a reader at that moment did. Restored on its own, the store finishes every transaction that moment had already decided — committed, or staging with every part landed, or aborted — and leaves one it had not decided standing and unseen, so the log taken after the snapshot still applies on top and finishes it.

What it costs, and when to use it

One majority round to each leader it writes, in parallel — a little more than twice what one write costs, because two leaders and the record must each be held rather than one. Measured on three processes on one Linux host, a transaction writing two shards led by two nodes committed at 2.4–2.7 times one write at the median. Across injected network distance, see what distance costs. Use it where records on two leaders must change together: a transfer between two accounts, an order and the stock it reserves. Where they need not, keep each transaction inside one leader's ranges — by placing related records in one shard, with PARTITION BY or a shared key prefix — and let it commit in one step.

Seeing it

INFO FOR NODE reports cluster.across: committed, aborted and in_doubt, counted by the node that drove each one, and pending records and transactions with_intents, as the settling round last sampled them. /metrics carries tessari_transactions_across_leaders_total{outcome}, tessari_transactions_pending and tessari_transactions_with_intents, and the console's cluster map shows the same counts.