Bounding the log

The log is not a logfile. The state of this store is a function of it, four different things replay it, and a record removed from it does not come back. Since 0.18.0-beta a serving node bounds it anyway, because a log nobody bounds grows until the disk is full — and the routine backup is now a snapshot, which does not need the history.

A serving node keeps the newest 100 000 records of each log and prunes the rest, once at start and then every ten seconds. Three things set the number, and the first one present wins:

INFO FOR NODE;              -- retain: 100000, retain_source: 'default'
DEFINE NODE RETAIN 500000 RECORDS;
DEFINE NODE RETAIN NONE;    -- keep the whole log, and remember that choice

retain reads null when the log is unbounded — NONE from a statement or the environment — and retain_source says who decided it. RETAIN NONE is stored as a choice, so a node told to keep everything keeps doing so after a restart whatever its environment says.

Upgrading a store that never set a number: it starts pruning at its first pass. If its history matters, take a snapshot first, or start it with TESSARIDB_RETAIN_RECORDS=none.

It is local, like ROLES and for the same reason

A disk budget describes this machine. The clause does not replicate, a restored backup does not inherit it, and each node of a cluster is set on its own. A node with a small disk and a node with a large one are allowed to disagree about this, and there is nothing to reconcile.

What a prune costs

Pruning is not reversible and the cost falls on whatever was still reading the part that went:

The refusal is a refusal rather than a short answer, and it says where the log now begins and what to do:

sequence 40 is below the start of this log, which begins at 51; the records that
answered there have been pruned, so this reader cannot catch up and needs a fresh
copy of the state: a follower node copies its leader's state by itself, and a
client re-reads what it follows and follows again from the current tail

A reader given a short answer instead would have no way to tell a log that ends there from a log that was cut there, which is the one distinction that decides whether to retry or to copy.

A follower that fell behind the window

A follower stopped for longer than its leader's window comes back below the leader's log start. It does not stay stuck: it copies the leader's state for its subscription over one peer connection — the leader holds its log for the length of the copy — installs it, removes what the leader no longer holds, and collects again from the positions the copy stood at. A copy that fails part-way leaves the node behind rather than damaged, and the next round tries again.

Where it stands is in INFO FOR NODE under cluster.upstream, on /metrics and on the console's cluster map:

/metrics carries the same as tessari_replica_state{state="…"} (the state the node is in reads 1), tessari_replica_copied_records, and on a leader tessari_follower_behind_records{node="…"} for each follower it serves.

The count is the whole bound, deliberately

A reader inside the window is safe because it is inside it. A reader further behind than the window is not protected, and is told so.

The alternative — holding the log down to whatever the slowest reader still needs — sounds safer and is the best-documented failure in this whole area: one stuck subscriber fills the disk while nothing anywhere is in an error state, and the store dies of a consumer nobody was watching. A floor that protects everything protects nothing, because the disk is finite and the laggard is not.

So the number in force is the number that holds. A follower that falls behind it copies; a client that falls behind it re-reads.

Two properties worth knowing before you choose a number

The last record always survives, whatever is asked for. A follower that is perfectly level is served by reading the record before the position it asks for, so a log with nothing left in it could not answer a healthy follower.

RETAIN 0 RECORDS is refused rather than clamped, and so is TESSARIDB_RETAIN_RECORDS=0. Keep nothing is the one request this cannot honour, and a setting that quietly runs as something other than what it says is worse than one that will not run.

Space comes back on the engine's schedule, not at the statement

A prune removes the records; the bytes return when the store next compacts that region. Nothing is lost by the delay — but a store watched immediately after a prune has not shrunk yet, and reading that as a failed prune is the mistake to avoid. The node says what it did:

pruned 50 log record(s) across 2 log(s) to the retained count

Choosing another number

100 000 records is a window, not a guarantee. The order of operations for setting your own:

1. Take snapshots on a schedule. Once a node prunes, a snapshot is the only whole backup it can take. See Backup and restore.
2. Measure how far behind your followers and subscribers actually get, from cluster.followers in INFO FOR NODE — behind is the figure, and it is counted in the same records this clause counts.
3. Choose a window comfortably above the worst of those, and set it — in the environment for every node of a deployment, or with the statement on one.
4. Watch the prune line in the node's log. A node that never prints one is a node whose log has not yet reached the window.