Leadership and failover

A node in a cluster writes under a leadership and at no other time. What makes a node part of a cluster is its catalog, not the roles it was given: a store whose membership rows name another node is in one, whatever it is declared to be.

From then on its writable role is what the operator wants and the lease is what it has.

Two refusals, and they say different things

this node is in a cluster and holds no leadership: it does not accept writes
until a majority grants it one …

this range is led by another node: write it at db-2.internal:9000.
This node refuses rather than forwarding on your behalf.
The node to expect is c81b…, leading under epoch 7

The first means wait. Nobody has been elected for this range and there is nowhere to send you.

The second means go there. A node that knows a leader names it rather than telling you to wait for a round that may never concern you. It is a routing answer rather than a failure: the write was correct, and there is a named place to take it. A node holding a perfectly live lease meets this refusal, which is the point — a leader over one namespace has no authority at all over another's. From 0.20.0-beta a client that greets with protocol minor 1 or later is not shown this text at all: the write comes back as a redirect it follows, and an HTTP request as a 307 (see where to go). The refusal is what a client older than that still reads.

Telling them apart matters because a client that retries on both will spin forever on the second and give up too early on the first.

A node whose leadership covers the whole store — which is every cluster that has elected a single leader — never meets the second one, because the whole store contains every namespace in it.

Which peers vote

A peer declared with the coordinating role is a voting member: a node that may write puts a ballot to every such peer when its lease approaches its fence, and leads for the next epoch if a majority grants it.

DEFINE REPLICA db_2 AT 'db-2.internal:9000'
    NODE '3f9a1c04b7e2489db5610af3d82c7e46'
    ROLES serving, coordinating;

The majority is a majority of the membership, and the membership includes this node. Three coordinating machines means a round is carried by two of them, one of which may be the candidate itself — so a cluster of three goes on electing after it loses one.

Every coordinating node may stand, not only the writable one, because a failover with one eligible candidate is not a failover. writable still says which machine the operator wants to lead and is what a voter weighs; it is no longer the only machine that can.

Which roles let a node stand is what the catalog says, not what it is currently doing. A leader that misses a round stops writing — that is the fence — but it goes on standing, because the role the operator wrote has not changed. A node the operator never made coordinating never stands at all, however quiet the leader gets. A placed range is the exception: only a node that may also take writes stands for it, because its leader is the only node that can write it.

A node with no peer declared is never fenced. A single machine has nobody to diverge from, so it takes no lease, and a store that never needed leadership does not acquire a new way to stop accepting writes — including one declared coordinating, because a deciding set of one elects itself.

A voter refuses a candidate whose log is behind its own

Opening the candidate set makes this necessary rather than merely tidy: a node holding less history could otherwise win a majority, lead, and silently drop every write it never received.

The comparison is the pair (the leadership that wrote the tail, the tail). A higher leadership wins outright, and the longer log decides only within one leadership — because a node that led an old epoch and then fell away can hold a higher sequence than the node carrying the history that actually won.

The refusal names the voter's own position, so a candidate can tell catch up and stand again from the history you hold is not the one that won.

A restarted voter cannot grant below what it already granted: it adopts the highest epoch it has seen, so a round it took part in before the restart is not one it can be talked into re-deciding.

The lease

INFO FOR NODE reports cluster.lease — the time left before this node stops accepting writes on its own.

It is null when no lease was ever granted, which is the ordinary state of a node standing alone, and null is a different answer from "0s". A node nobody made a leader is not a leader running out of time, and reporting it as one would leave every single-node deployment permanently at the value that is supposed to mean trouble.

When it is a duration it is the number to watch, because the dangerous state is exactly this at zero while the node is still taking writes. A leader that has lost the rest of the cluster does not know it has; it keeps accepting writes, and every one of them is a write the next leader will not have. The lease is the answer: leadership is held for a bounded time, it must be renewed to continue, and a holder that has not renewed refuses writes without being told to.

The figure counts down to the moment this node stops, which is deliberately earlier than the moment the cluster may hand the leadership to somebody else. The gap covers the difference in rate between two clocks nobody synchronised: if they coincided, a holder whose clock ran slightly slow would still be writing at the instant another node was told to start. So the number here is always the smaller, honest one.

It is measured on elapsed time and never on a calendar clock. A clock that steps backwards would extend a lease that should already have died, and a fence that an NTP adjustment can widen is not a fence.

tessari_lease_remaining_seconds on the metrics endpoint is the same number. There, a node holding no lease publishes no series at all rather than a zero, for the reason the null exists here: a series reading zero on every standalone node teaches whoever watches it to ignore the one reading that matters.

Losing a leader is noticed twice

Peers exchange a heartbeat on the link between them, which detects a lost leader faster than the awareness round that would otherwise find it. A healthy cluster stays quiet — the heartbeat is not a round of elections, and no campaign runs while a leader is renewing.

How long the cluster waits before it replaces a leader

The periods that decide when a leader counts as gone are a cluster-wide fact, so they are a statement and a replicated row rather than a file or a flag:

DEFINE FAILOVER
    AWARENESS 10s
    COLLECTION 10s
    ROUND 1s
    CAMPAIGN 1s
    LEASE 30s;

A configuration file cannot carry an agreement of this kind. It is an unreplicated claim about something every node has to agree on, so two nodes holding different files is not a conflict anything detects: each is internally consistent, each is confident, and the disagreement shows up only in the outcome — two nodes that disagree about when a lease has expired are two nodes that can both believe they may write, which is the split-brain the lease exists to prevent, arriving through the mechanism meant to prevent it. A command-line flag is the same claim with a shorter life.

Every clause is required. The five values are checked against one another rather than one at a time, so a statement naming three of them could only either mix new values with old ones under a single version, or perform a read-modify-write the operator cannot see. DEFINE NODE lets a clause be left out because it amends a row, and absence there means leave that field alone; this statement replaces a set.

What each period is for, and which direction it hurts in — the direction is the part worth knowing, because both ends of every one of these is a real failure:

Four relations are enforced, and a statement that breaks one is refused with the direction named rather than accepted:

• every period is at least 50 milliseconds;
• CAMPAIGN is at most twice ROUND;
• LEASE is greater than the write fence plus twice ROUND;
• COLLECTION is less than twice AWARENESS, which is the tightest staleness bound this node admits.

The refusals say which number to move:

`campaign` is 5s against a `round` of 1s, and it may not exceed 2s — twice the
round time …

Two things it deliberately does not let you set

The lease guard is not settable. It is the margin between the moment this node stops writing and the moment the cluster may hand the leadership to somebody else, and a fence an operator can set to zero is a fence somebody sets to zero on the day a lease refuses a write they wanted — which removes it rather than tuning it.

The staleness floor is not settable either, for the opposite reason: it is derived from AWARENESS, and a stored copy is a second answer that gets left behind the first time awareness moves.

Who may write it, and what orders two of them

The statement is a log record like any other, so it is written by the node holding the leadership and reaches every other node through the log. It carries the leadership it was written under, and which setting under that leadership it was, and a policy installs only if that pair is higher than the one already held.

Neither number is a clock, and that is the point — a clock is the one ordering two working nodes can disagree about. The leadership settles a partitioned former leader reconnecting with a policy of its own; the second number settles one leader setting the policy twice, which the leadership alone cannot tell apart.

INFO FOR NODE reports it under cluster.failover, with the pair that orders it. It is null when nobody has set one, which is a different statement from the defaults: a cluster nobody has configured runs the built-in periods — AWARENESS 1s, COLLECTION 1s, ROUND 200ms, CAMPAIGN 100ms, LEASE 800ms, which replace a lost leader in about a second — and reporting those as a set policy would make it impossible to see whether a policy ever arrived.

Desired role beside effective role

DEFINE NODE ROLES sets what this node is — immediately, locally, in the metadata a backup does not carry. That is the effective role.

A cluster needs the other one as well: what the node is supposed to be, written once by whoever administers the cluster and readable on every node. That is the desired role, and it is a membership row bound to a node:

DEFINE REPLICA db_1 AT 'db-1.internal:9000'
    NODE '9f2c4e1a70bb43d5a1c6e2f480937d55'
    ROLES serving, writable;

INFO FOR NODE reports both, and they may differ — that is not a fault to be reconciled away. The desired role is the instruction and the effective role is the state; a node that has been drained for maintenance is exactly the case where the two ought to disagree, and a report that showed one number would hide either the instruction or the truth.

Taking a node out of service

A node that has been drained is the case above, and there is a statement for it:

DEFINE NODE ROLES NONE;

The node keeps its data, its identity and its place in the membership. It stops answering clients and stops accepting writes. That is what draining is, and it is how a machine is taken out of service without being stopped — for a disk change, an upgrade, or an afternoon of being suspected.

NONE is a whole answer and not one of the roles. ROLES NONE, serving is refused rather than read as one of the two, because a statement naming both would have to decide which won, and every reading of it is somebody's reasonable expectation.

It exists because leaving the clause out means the opposite here. A clause omitted from DEFINE NODE leaves its field alone — that is what lets DEFINE NODE ENDPOINTS … change an address without disturbing the roles — so absence cannot also mean clear them. Without NONE the empty role set was a state the store could hold and no statement could ask for. A peer is declared rather than amended, so an absent ROLES already clears there and DEFINE REPLICA … ROLES NONE stays refused; declaring and amending are different acts, and the grammar keeps them different.

Draining the last node that serves is allowed. Nothing refuses it, because the reason to drain a node is usually that it is about to be stopped, and a rule that refused the last one would refuse exactly the case it was written for. What a drain costs is a question for whoever is watching the cluster, and INFO FOR NODE reports every peer's roles so it can be answered before the statement is sent rather than after.

On a node a membership row declares a role for, a drain is an override the next open discards. The desired role is the shared truth and a local word that outlived it is a disagreement nobody can see. To drain a bound node so that it stays drained, write the membership row instead.

What happens when a leader is lost

Concretely, with three nodes and one namespace replicated to two of them: the leader stops renewing, its lease runs down and it refuses writes before the cluster is entitled to replace it; the heartbeat notices; the remaining coordinating nodes run a round; one wins with a majority of the membership and takes the next epoch; and a client reading throughout is routed to a node that can answer.

The client's write is refused during the gap, and it is refused with holds no leadership — wait — rather than with an address that is not yet true.

When two histories claim one position

A fork is what the epoch exists to make visible. If two nodes both believed they were leading, they each wrote records at the same log positions, and a follower offered the second one refuses it:

log divergence at 812: this store holds epoch 3, and epoch 4 was offered

Three properties of that refusal are load-bearing. It happens at the first divergent record, named by sequence, rather than at some later position where the damage is already applied. It is not retryable — the same record will still be from the other branch, and the node re-bootstraps rather than retrying. And every occurrence increments a counter that is published, as tessari_log_divergences, because a fork that is refused correctly and never reported is a fork nobody investigates.

The sequence and the epoch are per range, not per store. Two ranges advance independently, and the refusal above still holds within a range — which is what makes two nodes leading two namespaces a normal arrangement rather than the failure this message describes.

The campaign is the only thing that produces an epoch. There is no other construction site anywhere in the engine, and that is checked structurally rather than believed: a second producer added anywhere fails the check. An epoch that could be minted outside a round would be a leadership nobody granted.