Reads, routing and staleness

A read on a cluster has two questions attached to it that a read on one node does not: which node answers, and how old that node's copy may be. Both are answered before the read runs, not after.

The reads below name four tables; on one node they answer as any read does:

DEFINE COLLECTION orders;
DEFINE COLLECTION events;
DEFINE COLLECTION accounts;
DEFINE COLLECTION prices;

STALENESS decides who may answer

SELECT * FROM orders STALENESS 30s;
SELECT * FROM events WHERE at > 0 STALENESS 5m;

STALENESS is optional and says how far behind the node answering this read is allowed to be. It is a candidate filter, never a marker: it does not ask to be told that an answer was stale, it says which nodes may answer at all.

That distinction is the whole design. A marker nobody is obliged to read is not a guarantee — a caller who forgets to check it gets the stale answer anyway, and the field that was supposed to protect them is the field that records they were not protected. It is also why a read that no node can satisfy is refused rather than quietly promoted to the one node that certainly can.

On a node standing alone the bound is satisfied rather than ignored — a node's own answer is never stale relative to itself.

There is a floor, and the refusal names it

A bound tighter than the interval at which this node learns anything about its peers is a promise nothing can check: it would be enforced against a picture whose own age exceeds the tolerance being compared to it. Such a bound is refused, and the refusal names the floor, because a caller told only that their bound was too tight cannot write a statement that would be accepted.

The floor is twice the awareness interval — one interval to learn something, and one more to notice that you did not.

STALENESS 0s, and any negative span, admit no node at all including the one being asked, so they can only ever refuse. That is a mistake in the statement and is caught where the statement is read.

It cannot be combined with VERSION

VERSION names one exact point in this store's history. A tolerance for how old that point may be is not a narrowing of it — it is a second answer to a question already answered. The two together are refused.

staleness is not a reserved word: a field may still be called staleness, and SELECT staleness FROM readings still reads it.

ANSWERED BY says where the answer must come from

SELECT * FROM accounts WHERE id = 1 ANSWERED BY LEADER;
SELECT * FROM prices ANSWERED BY ANY;

ANSWERED BY is optional and takes exactly two words: ANY, which is what a read means when it says nothing, and LEADER, which admits only the node that decides writes for these records.

Any other word is refused, and the refusal echoes what was written. The direction a guess would fail in is the unsafe one: somebody who wrote ANSWERED BY MASTER meant the leader, and a parser that shrugged and admitted any copy would answer the read they were being careful about from a follower, with nothing anywhere in an error state.

It is not a tighter STALENESS, and the difference is the whole point

A follower at zero lag is level, not authoritative. Being level a moment ago says nothing about a write committing right now, so no staleness bound expresses this must come from where writes are decided — and the tightest bound that might look like it, STALENESS 0s, is refused anyway, because it admits no node at all including the one being asked.

So the two are separate controls answering different questions, and they compose. A read may name both and is answered only where both hold:

SELECT * FROM accounts WHERE id = 1 STALENESS 30s ANSWERED BY LEADER;

Which node may answer is settled first. A node that may write is level with itself, so the leader satisfies every bound — deciding it first therefore never overturns the freshness decision, while the other order can: a follower inside the bound would be chosen, and the read that said it had to come from the leader would be answered by one that does not.

A node that cannot satisfy it says where to go

Like a staleness bound, this is a candidate filter, never a marker. A node that does not lead, and knows of a peer that claims to, answers with a redirect naming that peer — it does not fetch on your behalf, and it says so. The redirect carries the leadership that peer published, so a node that has since lost it can refuse the redirect on arrival rather than failing opaquely.

A node that does not lead and knows of no peer that does refuses:

this read asked to be answered by the leader: this node does not lead, and no
peer it has greeted says it does. Declare one with `DEFINE REPLICA … ROLES
writable`

The read said where it had to come from, and it is not served by coming from somewhere else and saying nothing.

Answered by the leader is not repeatable

This is the limit, and it ships with the clause rather than being discovered later. Two reads answered by the leader, with writes in between, legitimately differ — and neither is wrong. The clause says where the answer comes from; it does not hold the store still while you read it. Reads that must agree with one another belong in a transaction, which is one point in the store's history.

answered is not a reserved word — a field may still be called answered, and SELECT answered FROM tickets still reads it. The clause opens only when BY follows.

Where to go

When a request cannot be answered here and another node can answer it, the answer names where to go: the address, the node to expect there, and the epoch it leads under.

It reaches a client as a frame carrying a settled or transient discriminator, never as an error. A client can therefore tell a redirect from a refusal without parsing a message. Over HTTP it is a 307 with a Location — temporary-redirect and not 302, because only 307 promises that the method and body survive the redirect, and a 307 without a Location is not a redirect a client can act on.

A redirect is sent only when nothing in the script has committed. A client follows one by sending the whole script to the other node, so a script that already wrote something here is refused instead, and the part that ran stays written. A BEGIN … COMMIT refused for its leader rolled back at COMMIT, and is redirected whole.

A caller that cannot follow is carried

From 0.21.0-beta, a node with a peer link does not send a redirect to a caller that cannot follow one — an HTTP request, or a wire client built before redirects existed (protocol minor 0). It carries the request over the peer link to the node that can answer it and returns that node's answer as its own. A wire client of minor 1 or later keeps receiving the redirect, which costs no extra hop once it has learnt where the leader is.

• No password crosses the link. The request travels under an assertion the carrying node signs with its peer key: who asked, which node is asked, the script and its parameters, a single-use nonce and a 30-second expiry. The answering node checks the signature against the certificate that same connection proved, checks the account against its own catalog — a changed password or a dropped user no longer matches — and refuses an assertion it has seen before.
• One hop. A carried request is never carried again; a refusal naming yet another node comes back as that refusal (over HTTP a 409), and a hop that failed is a 502 naming why.
• Data statements only. A script that changes users, grants, membership or the failover policy, takes or restores a backup, or unseals a vault, is never carried; its caller gets the redirect or refusal it always got.
• The session goes with it: the namespace and database selected before the script ran, and its bound parameters.

A node with no peer link — a cluster run without peer credentials — has nothing to carry a request over, and answers as before: the refusal naming the address (this range is led by another node: write it at db-2.internal:9000) to an old wire client, and over HTTP a 307 with a Location, or a 409 where there is none.

A peer row's CLIENTS AT and HTTP AT say where a client reaches that node; a redirect names them. A row that names neither keeps redirects naming AT.

The clients follow it

Every client from 0.7.0 — Rust, Python, JavaScript, Go and Kotlin — follows a redirect inside the call that received it, by one algorithm the protocol states:

• at most three hops; a fourth redirect is a loop, and is an error of its own;
• never back in time: a redirect naming an older epoch than one already followed is refused;
• the session travels: before leaving, the client asks the node it is on which namespace and database the session selected, and selects them again on arrival — only names that are plain identifiers, so a name is never quoted into a statement;
• the node is checked: on arrival it asks which node it reached, and refuses to send the request to any node but the one the redirect named;
• settled moves the connection to the new node; transient answers the request there and leaves the connection where it was.

Both questions the client asks are one function, which any session may call:

RETURN session::context();

It answers { node, namespace, database } — the identity of the node this session is talking to, which is the identity a redirect carries, and the names the session selected, null where it selected none.

A JavaScript client over a WebSocket cannot dial another node from a browser, so it returns the redirect to the caller instead of following it.

A routing answer carries the epoch it was decided under, and a redirect naming an epoch older than the receiver already holds is refused with the newer epoch named. Without that, a redirect captured during one leadership and replayed during the next would send a write to a node that stopped leading — and nothing about the reply would say so.

The question is answered locally. A node says who leads a range from a view it materialised by applying the log, with no network call — so it can still answer while the peer link is down, which is exactly when a client most needs to be told where to go.

Two nodes, two namespaces

Two nodes may each lead a different namespace at the same time. Each accepts writes for its own and refuses a write for the other's, naming the node that leads it. Both statements are correct, both nodes are healthy, and there is nothing to fail over.

That is the arrangement the per-range epoch exists for, and it is the reason may this node write and may this node write here are two questions. See Leadership and failover.

What the leader knows about a follower it never calls

cluster.followers in INFO FOR NODE is not cluster.peers seen from another angle. A peer is what somebody declared; a follower is what has actually collected. A peer that has never asked this node for anything appears in the first list and not the second, and that gap is the most useful thing either list says.

A follower collects by asking, so the leader has handled every record that follower holds and needs no connection back to it to say how far it has got — which matters, because a leader that dialled out to measure lag would lose the measurement exactly when the follower became unreachable. From 0.21.0-beta the follower keeps that connection open and the leader sends each commit down it as the commit lands, rather than waiting to be asked again; the follower still dials, so the leader still needs no connection back.

Each row carries three numbers beside its position, and each is blind to a failure the others see:

quiet_for is time since the last collection, not the delay between a commit here and its application there. A follower that is level was current as of that long ago; one that is behind has not been trying for that long.

copy_age is the other question, and the two come apart on an idle node: a follower that is perfectly level goes on reporting a quiet_for that grows for as long as there is nothing to collect, while the copy it holds stays current. The node answers it by dating its own committed tail once per awareness interval and reading each follower's position against that timeline, so the figure is an upper bound — it overstates by at most one interval and never understates, which refuses a copy that might have been fine rather than serving one that might not be.

It is null, and not zero, when the copy predates everything this node has dated: a caller must read null as beyond every bound. A node that has just started reports null for every follower until its first interval has passed.

Three behaviours are deliberate and each removes a way of being lied to:

• An empty collection still counts. A follower that is level asks and receives nothing, and reading that as silence would report the healthiest follower there is as absent.
• A follower asking from an earlier position is recorded at the earlier position, not at its best-ever one. Asking backwards is what recovering from a divergence looks like, and a high-water mark would hide it.
• The list is not persisted. A restarted node reports no followers until one asks it for something, because a stored row saying a follower reached position 812 four seconds ago would outlive the relationship it describes.

Where a follower collects from

A follower collects from whichever peer says it may write right now, not from the row you marked writable. In a cluster that can fail over, every coordinating node is also declared writable — otherwise the node that wins a round could not take writes — so the declaration cannot pick an upstream on its own. Each node therefore follows the declared peer whose most recent greeting says it may write, preferring the newer leadership when two of them do.

A node that has just started follows nobody until its first greeting round lands, which is one awareness interval.

One answer is bounded

A follower asking for a large stretch of log receives a bounded answer and is told it stopped early, and a second call resumes from where the first ended. A collection that returned everything asked for would let one catching-up follower decide how much memory the leader uses.