Who leads what

A cluster begins with one line — the store line — and its leader writes everything. To have two nodes take writes at once, a range is placed: a member row says which range its node stands to lead, and that range becomes a line of its own, with its own leader, epochs and lease. This page is how a placement is made, moved, preferred, given back, and — when you ask for it — kept even by the cluster itself. The last three arrived in 0.24.0-beta.

Placing a range: LEADS

DEFINE REPLICA b AT 'db-2.internal:9000'
    NODE '9f2c4e1a70bb43d5a1c6e2f480937d55'
    ROLES serving, writable, coordinating
    REPLICATES STORE
    LEADS SHARD prod.shop.orders 2;

The range is a NAMESPACE, a DATABASE or a SHARD, spelled as REPLICATES spells it; LEADS STORE is refused, and so is a shard the table does not have. A row places one range. Several rows naming one range make several candidates for it — that is how a range gets a second node to fail over to.

A placed range is its own election, run exactly as the store's is (Leadership and failover), so the node leading a shard and the node leading the store can be different nodes, and writes to the two are taken by both at once. Every node collects a placed range from whichever node leads it.

The placement takes the range away from the store's leader at once. From the moment the row commits, a write into the range is refused NoLeadershipYet until a candidate is elected, and afterwards sent to the range's leader. The store's leader is deliberately not a fallback for a placed range whose leader is gone — writing it would make two writers of one range.

Only a node holding the whole store leads the store. The store's leader writes every range no row places and collects from nobody, so a node whose own row says REPLICATES SHARD, DATABASE or NAMESPACE stands only for the range its row places. It still votes in the store's election.

INFO FOR NODE reports each peer's placement as leads, in the clause's own spelling, with preferred and releasing beside it.

Moving a placement

To drain a node, place another row on its range and take the first away:

ALTER REPLICA c LEADS SHARD prod.shop.orders 2;
ALTER REPLICA b LEADS NONE;

b stops standing for the range once it has applied the change, and every voter that has applied it refuses b a renewal, so b's lease runs out and c is elected under a later epoch. Writes into the range are taken by b until then, refused while c is elected, and taken by c afterwards. Measured on three local processes with a writer running throughout, the move completed 11–20 seconds after it committed and no write was taken for at most 6.2 seconds — one collection of the store's log plus one 10-second lease.

Giving a range back to the store line

LEADS NONE on the last row placing a range gives the range back. It does not drop the placement — dropping it would let the store's leader write the range at once while the range's own leader went on writing under its lease. The row is marked releasing instead, and the hand-back runs in four steps:

1. the range stays carved out of the store line on every node, so nobody but its own line writes it;
2. the releasing node stops standing for it, and voters refuse it a renewal, so its lease runs out;
3. the store line's leader — when it is placed on no range of its own — stands for the range on the range's line and is elected once that lease has lapsed;
4. leading both, it folds the placement away, and the range is the store line's again — led by the same node, so no instant has two writers.

Writes into the range are refused for about a lease and an election while this happens, and land on the store's leader afterwards. Measured on three local processes with a writer running throughout, the store's leader took the range about 3.7 seconds after the release, and every write the writer was told was taken was held exactly once. ALTER REPLICA b LEADS <the same range> withdraws a release that has not folded yet; placing another row on the range makes it an ordinary move.

• PlacementCannotBeDropped — DROP REPLICA on the last row placing a range, or moving it to another range. Give the range back with LEADS NONE first, or place another row on it.

Preferring one candidate: PREFERRED

When several rows place one range, the election decides which leads, and any of them may win. Written after the placement, PREFERRED names the one that should:

ALTER REPLICA c LEADS SHARD prod.shop.orders 2 PREFERRED;

A leader that is not preferred, hearing a preferred candidate that is caught up with it on the range's line, stops renewing; its lease runs out and the preferred candidate is elected — the same hand-over a moved placement takes. It hands over only to a candidate that is caught up and audible, stays off the range's line for four leases afterwards so it does not win it straight back, and hands over at most once every 30 seconds, so a preferred node that keeps failing cannot have the range thrown back and forth. Measured on three local processes, the preferred candidate took the range about 2.6 seconds after it was placed. Restating the placement without the word clears it.

Use it to keep a range on the node nearest its writers, or to bring a range home after its preferred node comes back from maintenance.

Evening out who leads what: BALANCE LEADERSHIPS

A failover leaves leaderships wherever the election put them, and over time one node can end up leading the store line and the range its own row places while another voter leads nothing. One clause on the failover policy asks the store's leader to correct that:

DEFINE FAILOVER AWARENESS 1s COLLECTION 1s ROUND 200ms CAMPAIGN 100ms LEASE 800ms
    BALANCE LEADERSHIPS;

With it, the store's leader counts the lines each node leads and moves a placement — exactly as ALTER REPLICA idle LEADS <range>; ALTER REPLICA busy LEADS NONE; would, in one transaction through the log:

• only when the busiest node leads at least two lines more than the target;
• only onto a voter — a coordinating peer bound to its node, subscribed to a reach that holds the range, and placed on nothing else;
• only the busiest node's own placement, and only while it leads it, never a preferred placement and never one being given back;
• at most once per two LEASE periods.

Each move is a line in the store leader's log (balanced leaderships: moved the placement of …) and a change to the rows INFO FOR NODE reports as leads; /metrics counts them as tessari_balancer_moves_total. DEFINE FAILOVER replaces the whole policy, so a later one without the clause turns the balancer off, and cluster.failover.balance_leaderships says which.

The balancer moves leaderships, never data: it never creates a placement, and a range nobody placed stays with the store line. Splitting a table into more shards is Splitting a table into shards.

Writing two leaders' ranges in one transaction

A transaction that writes ranges two different nodes lead is refused SpansLeaderships, naming both — unless it asks to commit across them with COMMIT ACROSS LEADERS. See Transactions across leaders.