Serving

$ tessaridb ./data --serve 0.0.0.0:7654 --http 0.0.0.0:8000

One process, two surfaces. --serve is the wire protocol and --http is HTTP; either may be given alone, and giving both is one node answering on both.

The three probes

None of the three needs a credential, and that is deliberate rather than an oversight: a probe that needs configuring is a probe nobody configures. A load balancer's health check is written once by somebody who does not have your passwords, and a store that demanded one there would be a store whose health check gets disabled.

/metrics answers anybody, and says more to a caller that signs in. Given the same credential /script takes, the scrape adds a series per topic that caller may read — read through the caller's own session, so grants decide which topics appear:

$ curl -s -u ops:… http://localhost:8000/metrics | grep tessari_topic_group_lag
tessari_topic_group_lag{namespace="shop",database="orders",topic="mail",group="mailer"} 3

Without a credential no topic or group is named, because a name is schema and the probe answers strangers. A credential that is refused is answered 401 rather than quietly given the anonymous scrape, so a monitor with a wrong password finds out.

On a node that belongs to a cluster, the anonymous scrape also leaves out every peer, member, address and leadership figure (from 0.21.0-beta): a probe that answers strangers should not draw them the cluster. Those series, and tessari_tls_certificate_expires_seconds{surface} — when each certificate the node presents expires — are shown to a caller who may read INFO FOR NODE.

/health and /ready are different questions. A node that is alive but draining answers 200 to the first and 503 to the second, which is exactly the pair a rolling restart needs.

There is a fourth, for a store this process opened rather than one it serves:

$ tessaridb ./data --health
well — committed to sequence 0

Exit code zero when it is well, non-zero when it is not, so it stands in a container healthcheck or a cron line without anything parsing its output.

Draining

A node's roles are a statement, not a configuration file:

DEFINE NODE ROLES serving, writable ENDPOINTS 'here:9000';

Take writable away and the node keeps answering reads and stops taking writes. With a table already there to read:

DEFINE COLLECTION node;
CREATE node = { name: 'ada' };
DEFINE NODE ROLES serving;

…a write is now refused:

CREATE node = { name: 'grace' };

NotWritable. Reads carry on:

SELECT * FROM node;

And the statement that gives the role back is not itself a write, which is what stops draining from being a one-way door:

DEFINE NODE ROLES serving, writable;

A statement that sets nothing is refused rather than read as "clear everything":

DEFINE NODE;

UnexpectedToken. And a role the store does not know is refused rather than stored for later:

DEFINE NODE ROLES leader;

Unknown. A configuration system that accepts a word it does not understand is one where a typo is discovered by an outage.

The store knows three role words — serving, writable and coordinating. The third is the one that decides whether this node may stand for leadership, and a node in a cluster that was never declared coordinating stops accepting writes and never starts again. On a node standing alone it changes nothing. See Adding a node.

Read both halves back with one statement:

INFO FOR NODE;

Stopping

Send SIGTERM — which is what docker stop, systemctl stop and Ctrl-C all send — and the node shuts down in stages rather than at once:

1. it stops saying it is ready, so a load balancer takes it out of rotation;
2. it stops accepting new connections;
3. it lets the work in flight finish;
4. it closes the store.

The order is the point. Refusing connections and being unwilling to serve are two different things, and doing them in the wrong order drops requests that a balancer was still sending. A measured stop of a live deployment took 5.6 seconds for the node and 0.9 for the API in front of it, both exiting zero — almost all of the node's time being the deliberate drain rather than the close.

Give a supervisor a grace period longer than your drain. A SIGKILL partway through is not a corruption — the store is recovered from its log — but it is an in-flight request that never got an answer.

What one node holds open

Each surface counts two things, because holding a connection and running a statement no longer cost the same. A held connection is a task on the node's one runtime — a few kilobytes of buffers, not a thread — so an idle client, or a subscriber waiting for the next change, costs almost nothing: 10 000 subscribers were measured held on 14 threads. What is bounded tightly is the work inside the store.

Neither ceiling is configurable, and each surface has its own, so a flood of one cannot take the places the other needs to answer a health check. A process that may really hold that many connections needs its open-file limit above them.

The wire surface has one more deadline. A connection that is accepted and then says nothing would hold its place at the door for as long as it liked — a denial of service that costs the caller a single socket — so a freshly accepted connection has 10 seconds to send its greeting. It applies to the greeting only, never to the session that follows: a long-lived idle client is not disconnected.

Where the store lives

One directory, given as an argument. There is no configuration file to find, no include path and no defaults directory, which means a node that started is a node running the settings you can see on the command line that started it.

The exception is the two things that describe this machine — its roles and its endpoints — which are statements because they must not travel in a backup. See Adding a node.

Two settings a serving node takes

$ tessaridb ./data --serve 0.0.0.0:9080 --http 0.0.0.0:8000 \
    --backup-dir ./backups --unseal-for 1h

--backup-dir is the folder BACKUP … TO '<name>' and RESTORE SCRIPT FROM use; without it every TO is refused rather than written somewhere nobody chose — see Backup and restore. --unseal-for is how long an unseal lasts before the store seals itself, ten minutes unless TESSARIDB_UNSEAL_FOR or this flag says otherwise; the flag wins, and zero is refused at start. Both are refused on a command that serves nothing.

Logs

The node writes to standard error and nothing else. There is no log file to rotate, because the process's output is the supervisor's problem and every supervisor already solves it. In a container, set a size cap: the default json-file driver never rotates on its own, and a store that filled a disk with its own log is a bad way to learn that.

The published image

It serves both surfaces by default, turns either off through its address variable, declares the first user from the environment so a node that came up unattended is not left open, and carries a health check that asks /health rather than the accept queue. In a container has the ports, every variable, and the volume's ownership.