Adding a node

A node's configuration is in the language, not in a file beside it. There are two statements, they look similar, and the difference between them is the whole design:

The three roles

DEFINE NODE ROLES serving, writable, coordinating ENDPOINTS 'db-1.internal:9000';

The store knows exactly three role words, and each answers a different question:

coordinating is the one that is easy to leave off and expensive to leave off. A machine declared serving, writable with a read-only replica beside it is in a cluster, so it must hold a leadership to write — and nothing stands for one unless its own roles say coordinating. It therefore stops accepting writes and never starts again. The refusal names the remedy rather than only the state:

this node is in a cluster and holds no leadership: it does not accept writes
until a majority grants it one. A node that should be leading needs the
coordinating role (DEFINE NODE ROLES ... coordinating) so that it stands for one

A role the store does not know is refused rather than stored for later:

DEFINE NODE ROLES leader;

Unknown. A configuration system that accepts a word it does not understand is one where a typo is discovered by an outage.

Either clause may stand alone, so a role change does not have to restate an address, and a clause left out leaves its field alone — setting the endpoints is not a silent way to strip a node of its roles. What a clause does name replaces what was there.

Starting a cluster from nothing

Declare every peer in one transaction. From the moment the first DEFINE REPLICA naming another node commits, this node is in a cluster and refuses every local write until a majority grants it a leadership — including the DEFINE REPLICA statements that describe the rest of the cluster. A statement-at-a-time script therefore declares one peer and is refused for the second. One transaction is judged once, against a catalog that still names nobody:

BEGIN;
DEFINE REPLICA second AT 'db-2.internal:9000'
    NODE '9f2c4e1a70bb43d5a1c6e2f480937d55'
    ROLES serving, coordinating REPLICATES STORE;
DEFINE REPLICA third AT 'db-3.internal:9000'
    NODE 'c81b0f37a4e94a6f8d2e5417b90c3f26'
    ROLES serving, coordinating REPLICATES STORE;
COMMIT;
DEFINE NODE ROLES serving, writable, coordinating;

Each row carries its subscription: a peer row with no REPLICATES is subscribed to nothing, and that node stays up, greets, and never holds a copy.

DEFINE NODE ROLES is not part of the transaction and does not need to be: what a node is for is written to the machine rather than to the log, so it takes effect immediately and is never refused by the leadership gate. Put it last anyway — it is the statement that makes the node stand for elections, and there is nothing to stand for until the peers are declared.

Where a client reaches a peer

AT is where peers reach a node. When clients reach it somewhere else — a public name, another port, an HTTPS front — the row says so:

DEFINE REPLICA second AT 'db-2.internal:9180'
    CLIENTS AT 'db-2.example:9080' HTTP AT 'https://db-2.example:8000'
    NODE '9f2c4e1a70bb43d5a1c6e2f480937d55' ROLES serving, coordinating;

CLIENTS AT is the wire address and HTTP AT the base an HTTP 307 puts in front of the request's path. Both are optional and replicate with the row, so a node can say where to go from what it has applied, with the peer link down — a redirect names them. A row naming neither keeps redirects naming AT. A member is amended in place with ALTER REPLICA second CLIENTS AT 'db-2.example:9080' (or CLIENTS NONE), one clause at a time, keeping the node it is bound to and its subscription. INFO FOR NODE reports both for each peer, null when unsaid.

Where a node runs: its region

A row may also name the region its node runs in. It is unset until somebody says it, and set or cleared one clause at a time like the rest:

DEFINE REPLICA third AT 'db-3.internal:9180'
    NODE '4b7d0e2c91aa4f68b3e5d1c7a2f09e64' ROLES serving, coordinating
    REGION 'eu';
ALTER REPLICA third REGION 'us';
ALTER REPLICA third REGION NONE;

The region replicates with the row, INFO FOR NODE reports it for each peer (null when unsaid) and the console shows it on the peer's card. Today it is what ACKNOWLEDGE LOCAL MAJORITY counts — a majority of the voters in the leader's region; it is also the name a cluster organised by region will be built on.

A peer is named, and the name is what makes it one peer

second declared above is one peer. Declaring it again at another address is not a second peer:

DEFINE REPLICA second AT 'elsewhere:9001';

NameTaken — that is a second address for one peer, not a second peer. A re-runnable start-up script says so out loud:

DEFINE REPLICA IF NOT EXISTS second AT 'there:9001';

ROLES on a peer is optional, and leaving it off means no roles rather than every role. A peer nobody has said takes writes does not take them. That is the safe absence: an operator who forgot the clause gets a refusal naming it, where the opposite default would send a write to a node nobody said could take one. It is spelled exactly as DEFINE NODE's is, because it is the same membership field seen from the other side, and two spellings for one set of words would be two things to keep in step.

Adding a node to a cluster that is already running

Everything above configures a cluster by writing to every node before any of them starts. That is not how a node joins one that is already up, and the difference is that the newcomer writes nothing about the membership at all.

Two statements, one on each side. On the cluster, name the newcomer and grant it the log:

DEFINE REPLICA third AT 'db-3.internal:9000'
    NODE 'c41d8f2b60ae47f5b9c3e1a780d24e66'
    ROLES serving
    REPLICATES STORE;

On the newcomer, say what it is for, and nothing else:

DEFINE NODE ROLES serving;

Then start it with one extra flag naming a node already in the cluster:

tessaridb /var/lib/tessaridb --serve 0.0.0.0:9080 \
    --cluster-credential db-3.pem --cluster-key db-3.key \
    --cluster-authority cluster-ca.pem \
    --cluster-address db-3.internal:9000 \
    --seed 9f2c4e1a70bb43d5a1c6e2f480937d55@db-1.internal:9000

A seed names a node as well as an address. Peers prove themselves to each other with a credential issued for the node's own identifier, and the handshake will not complete against any other node — so an address on its own is not something a node can dial. The identifier is the one INFO FOR NODE prints.

The newcomer declares no peers of its own, and that is the point. A DEFINE REPLICA is a catalog write and therefore a record in the log, so a cluster whose nodes were each told their own membership before starting holds several logs carrying different records at the same positions. A newcomer that writes no membership row cannot produce that, and it does not need to: the membership reaches it through the same replication as everything else.

The seed is read only while the newcomer's catalog names no peer other than itself, and the catalog takes over the moment it names somebody else. A node that has joined therefore finds its way back after a restart without the seed flag — the membership it collected is in its catalog now.

The newcomer bootstraps from a live leader. Nothing is stopped and no snapshot is taken by hand: it replays the log from origin, and the keys it ends up holding are byte-identical to the leader's.

A node that already holds data of its own is refused

The newcomer above is a fresh store, and that is the ordinary case: the machine you add to gain capacity has nothing on it yet. A node that already holds namespaces of its own is a different situation, and it is refused before it dials:

this node holds namespaces of its own — `research` — and a cluster allocates
namespace ids from its own counter, so collecting would file this node's records
under the cluster's namespace, with its name, schema, replication class and
grants …

Nothing would be deleted, and that is what makes it dangerous. A namespace is the root of the address a record lives at — (namespace, database, table, id) — so what a join would replace is the definition at that address, not the records under it. Every one of those records would then be read through somebody else's name, schema, replication class and grants, with the row count unchanged, no error raised, and nothing in the log to find later.

There is no join anyway switch, and its absence is deliberate: an override that outlives the join it authorised is a standing permission to reinterpret whatever this node holds next time. The way through is to perform the removal rather than permit it, with a statement that already exists:

DROP NAMESPACE research;

DROP NAMESPACE refuses while a database is still under it, and DROP DATABASE refuses while a table is still under one, so you are walked down your own tree and name every object on the way. A store with no namespace of its own has no address left to reinterpret, and the join then proceeds with nothing remembered and no flag anywhere.

One step is easy to trip over. A node configured to join is usually declared ROLES serving, which holds no write authority, so the DROP is refused on the one node that needs it. DEFINE NODE is a local write and is always allowed, so:

DEFINE NODE ROLES writable;
DROP NAMESPACE research;
DEFINE NODE ROLES serving;

Keeping the data is not offered. An in-place merge would mean relocating every record, index entry, field and grant the node holds — a migration of the whole store, and still a silent rewrite of your data. If the records matter, re-insert them through the cluster's own tenancy after the node has joined.

What a peer may collect

Proving who you are at the door and being allowed to take something are two different permissions. What a peer may take is a subscription, written on its own row:

DEFINE REPLICA db_2 AT 'db-2.internal:9000'
    NODE '3f9a1c04b7e2489db5610af3d82c7e46'
    REPLICATES NAMESPACE prod;

REPLICATES takes the same three spellings a grant takes — STORE, NAMESPACE <name>, DATABASE <namespace>.<database> — because a subscription is a read grant over the addresses it names.

Without the clause a peer is subscribed to nothing, and that is the refusal rather than an oversight. A node asking for records it was never granted is told so in one sentence naming the statement that would grant them. It is not answered with an empty collection, because you may not ask and you are level must never look alike.

The authority behind it is a store-only kind called replicate. It cannot be carried by a namespace-reach grant — the reach of a subscription is decided by the row, and a kind that a namespace owner could hold would let a tenancy admit its own follower. See Authority.

What a subscription hands over is more than records. The log is a stream of mutations and the identity class is in it, so a subscription carries the users, their credential hashes and their grants inside its reach. REPLICATES STORE therefore hands over every tenancy's, which is why it is something an operator writes deliberately and never a default.

Approving the node that joins

A row that grants a subscription is bound to a node by the operator, never by whichever machine dials first. Three ways, chosen per row (from 0.21.0-beta):

DEFINE REPLICA db_4 AT 'db-4.internal:9000' NODE '3f9a1c04-b7e2-489d-b561-0af3d82c7e46'
    ROLES serving REPLICATES STORE;

DEFINE REPLICA db_4 AT 'db-4.internal:9000' ROLES serving REPLICATES STORE
    FINGERPRINT '0f1e2d3c4b5a69788796a5b4c3d2e1f00f1e2d3c4b5a69788796a5b4c3d2e1f0';

DEFINE REPLICA db_4 AT 'db-4.internal:9000' ROLES serving REPLICATES STORE;
CREATE JOIN TOKEN FOR REPLICA db_4 EXPIRES 10m;

• NODE names the newcomer's identity, read off it with INFO FOR NODE.
• FINGERPRINT pins the SHA-256 of the one peer certificate allowed to bind the row. Read it off the newcomer the same way: INFO FOR NODE lists under certificates each certificate the node presents — its surface (peers or clients), its fingerprint, and when it expires.
• CREATE JOIN TOKEN answers a token once, 64 hexadecimal digits; the row keeps only its digest and when it stops binding. EXPIRES is required, because a token nobody gave a life to binds for as long as nobody remembers it. Start the newcomer with --join-token <token> (or TESSARIDB_JOIN_TOKEN) beside its --seed; it offers the token to the seed every round until a row names it, and the node that may write binds the row and spends the token in one transaction. A second token for the row replaces the first.

A row naming none of the three waits — it grants nothing until a token binds it. The binding writes the node id and nothing else: the address, the roles and the subscription stay exactly as they were written, so nothing a greeting carries can change what the row grants. INFO FOR NODE shows a row's fingerprint and, while a token waits, join_expires_ms — never the digest. Every binding is recorded in the administration trail as BIND or JOIN.

A node dropped is never admitted again. DROP REPLICA of a row that named a node also records the node as removed: its greeting is refused at the handshake whatever certificate it presents, no row binds it, and a DEFINE REPLICA … NODE naming it is refused with NodeTombstoned. A machine coming back is wiped and joins under the identity it then has. So a row is not amended by dropping and declaring it again — ALTER REPLICA <name> ROLES … changes a member in place. INFO FOR NODE lists the removed nodes under cluster.tombstoned.

The link between nodes

Peers speak a binary protocol on their own port, distinct from the one clients use, and the connection is mutually authenticated TLS. Each side presents a certificate issued for its node identifier by the cluster's authority, and a node presenting a client certificate cannot join as a peer — the two roles are different certificate roles and the wrong one is refused at the handshake. Renewing those certificates, revoking one, and what an expiry does to a link already open are in Encryption in transit.

Reading it back

INFO FOR NODE;

Both halves, in one answer, as two named groups rather than one flat object: the flat fields are this machine and would not follow a backup, everything under cluster is the topology and would. Flattening them would make that a thing you have to remember, and the day it is remembered wrongly is the day last night's backup goes onto a fresh machine and two processes claim one identity.

{"id": "9f2c…", "roles": ["serving", "writable"],
 "endpoints": ["db-1.internal:9000"],
 "cluster": {"peers": [{"name": "second", "endpoint": "db-2.internal:9000",
                        "roles": ["serving"], "node": null}],
             "desired": ["serving", "writable"],
             "followers": [{"node": "4b81…", "sequence": 812, "behind": 4,
                            "quiet_for": "2s143ms", "copy_age": "11s"}],
             "lease": "58s"}}

It also reports version and build, which answer different questions: version is three ordered numbers and is what an upgrade compares, and build is what this binary actually is, pre-release suffix included. On a final release the two read the same, and the difference only appears when there is one — which is exactly when somebody needs to see it.

roles is what this node may do right now: the roles it adopted, without writable while it holds no lease on the store line. A member following the store leader therefore reports serving, coordinating although its row grants writable, and a range placed on it is still written there. cluster.desired is what its row grants; the node adopts it when it next starts, so a node that joined under DEFINE NODE ROLES serving keeps that until its first restart.

peers is what somebody declared; followers is what has actually collected. They are read in Reads, routing and staleness.

Why identity must not travel

Consider a node restored from another node's backup. If identity travelled in the log, it would inherit writable and accept writes it is supposed to forward, and every peer told to reach it would be given the original's address and reach the original. Neither failure announces itself.

And the mirror case: if the topology did not travel, a restored node would come up knowing about no peers at all, and a cluster rebuilt from a backup would be a single node that believes it is the whole thing.

So a backup must carry the peers and must not carry the identity, and neither test is the criterion on its own — a design that got one right and the other wrong would pass half the checks somebody thought to write.

Configuring a node is done on it

There is no DEFINE NODE <other> …, and its absence is a decision. A statement that reached across would be a second mechanism for something the topology already decides, and the two would disagree the first time a node was unreachable while its row said otherwise.

Taking writable away is how a node is drained, and it stays readable throughout — that is an operation on a running node and is covered in Serving.