Backup and restore

The store keeps no state that is not derived from its log. That one property decides this entire page: a copy of the log is a complete backup, and a restore is a replay.

Three backups, for three jobs

A store is backed up in one of three forms. The log is the one this page began with; the other two are the store's current state, sized by what it holds rather than by how often it was written. From 0.18.0-beta the snapshot is the default, because a serving node now bounds its log and a pruned log can no longer be copied whole.

The difference in size is the history. One record written ten thousand times is ten thousand records in the log and one in a snapshot — measured, 1 641 008 bytes against 1 100.

From a running node

BACKUP is a statement, so nothing has to stop for it — writes keep landing while it reads, and a write that lands while it reads is not in the file:

DEFINE NAMESPACE prod;
USE NAMESPACE prod;
DEFINE DATABASE orders;
USE DATABASE orders;
BACKUP;

It answers with a snapshot as bytes. The same thing named, and the other two forms:

BACKUP STATE;
BACKUP LOG;
BACKUP SCRIPT;

The log answers its whole history; incremental is a bound on it, and naming a position is enough to ask for the log:

BACKUP FROM 2;

Everything at or after that sequence. FROM takes a sequence, and says so when it does not get one:

BACKUP FROM soon;

UnexpectedToken. A snapshot and a script take no FROM, because each is a moment rather than a range of a log:

BACKUP STATE FROM 2;

UnexpectedToken.

BACKUP is the one statement whose subject is the store rather than the selected namespace, which is why every form of it needs an owner: there is no table for a grant to name.

Over HTTP the same thing is GET /backup — a snapshot — with ?as=log for the log, ?from=<n> for the log from a position and ?as=script for the script. The node writes a snapshot to its temporary folder and sends it from there with its length declared, so its memory does not grow with the store: measured, 28.5 MB at the peak for an 87 MB snapshot, against 553 MB for the same snapshot answered as a statement's value over POST /script. For a large store, use GET /backup or BACKUP … TO, and leave the node's temporary folder room for one snapshot.

Over the wire one answer is one frame, and a frame carries at most 16 MiB. A BACKUP whose answer is larger — a snapshot of a store that has grown past it — is refused as AnswerTooLarge, naming its size, and the connection goes on to the next request. Up to 0.33.4-beta the node dropped the connection instead, which a client reported as the connection ended mid-frame. From 0.34.0-beta (protocol minor 5) a client can ask for the snapshot to be streamed: the node sends it in parts of 1 MiB ahead of the answer, neither side holds the whole file, and the answer is then its summary — { form, records, version }. Only the snapshot streams; the log and the script are still answered whole.

A part of the store

A snapshot can be of one place:

BACKUP STATE OF NAMESPACE prod;
BACKUP OF prod.orders;

It carries what a follower subscribed at that place is given: the place's records, the catalog that defines it, and the store's users and grants — so a store restored from it is as closed as the one it was taken of. A database's sibling databases come with their definitions and none of their records. Two places are two snapshots, and a list is refused saying so.

A script can be of several:

BACKUP SCRIPT OF NAMESPACE prod, prod.orders;

OF names what a script carries — NAMESPACE x for a namespace and every database in it, DATABASE x.y or the bare x.y for one database. It carries their declarations, their records, their indexes and the analyzers their fields use (each IF NOT EXISTS, so a store that already declares one of that name keeps its own). It carries no users and no grants — they belong to the store, not to a namespace — and its header says so and names the places it holds. A place the store does not hold is refused as Unknown.

A log of a part is refused: a log of one database holds its records and none of the definitions of the namespace and database it lives in, so it would restore nowhere on its own.

BACKUP LOG OF NAMESPACE prod;

UnexpectedToken.

On a cluster

A snapshot is taken on any node that holds the whole place — a follower included, which keeps the load off the leader. It is that node's own applied state, and the file records where each log stood. A database whose split table is led by several nodes is backed up complete from a node that holds every shard's log; each shard is consistent at its own recorded position, and there is no single version across writers — the same statement a gathered read makes.

A node that holds only part of the place — some shards of a split table — refuses with NotHeldHere, naming the table and the shards it lacks. There is no gathered backup: a file assembled from several leaders is not one state of anything.

Written by the node, into its backup folder

BACKUP TO 'weekly/state.tessarisnap';
BACKUP LOG TO 'nightly.tessarilog';
BACKUP SCRIPT TO 'dump.tessariql';

TO asks the node to write the file itself rather than answer with it, into the backup folder it was started with — --backup-dir <folder>; in the container image TESSARIDB_BACKUP_DIR passes it, and is set to /var/lib/tessaridb/backups inside its volume. It is what the console's Backup tab runs. The answer says where the file landed and how large it is: { path, bytes, form }.

The name is a file inside that folder, and the node decides whether it stays there:

• it is relative and made of plain parts — an absolute path, a .. or a . is refused as BackupNameRefused, before anything is written;
• subfolders it names are made one at a time, and one that is already a link or a file is refused rather than followed out of the folder;
• a file already at that name is never replaced — BackupExists (409), because the file it would replace is most likely an earlier backup;
• a node started without a folder refuses every TO as NoBackupFolder (409) rather than writing somewhere nobody chose.

The bytes go to <name>.partial, are synced, read back through the verifier where they were written (a log and a snapshot; a script has none), and only then renamed into place — so the folder never holds a file that reads as a backup and is not one. The same owner rule applies as for BACKUP itself.

From the command line

For a store this process can open directly:

$ tessaridb ./data --backup ./monday.tessarisnap
$ tessaridb ./data --backup ./whole.tessarilog --from 1
$ tessaridb ./data --backup ./since.tessarilog --from 4200
$ tessaridb ./data --dump ./monday.tessariql

--backup writes a snapshot unless --from names a position, which asks for the log — --from 1 is the whole log. --snapshot is the same snapshot by its own name.

For a running node, name it with --at (from 0.34.0-beta):

$ tessaridb --at db-1.internal:9000 --backup ./monday.tessarisnap

The snapshot streams over the connection whatever its size and is written aside and renamed once the node has answered. It needs an owner's sign-in like any BACKUP; --from and --dump are still refused beside --at.

Each writes to a .partial file beside the one it was asked for and moves it into place only once it is whole, so a backup that fails leaves nothing at its destination — not an empty file, and not the half of one that a restore would read as a backup.

Checking one

$ tessaridb --verify ./monday.tessarilog
$ tessaridb --verify ./monday.tessarisnap

Note what is missing from that line: a store. --verify reads the file and says what it holds, changing nothing and needing nowhere to put it. That is the whole reason it exists — a backup that can only be checked by restoring it is a backup nobody checks, and the ones nobody checks are found to be empty on the day they are needed.

Check the file where it is written and before it is kept. A script that verifies after moving the file into place has a window in which the backup directory holds something unreadable.

Restoring

$ tessaridb ./fresh --restore ./monday.tessarilog
$ tessaridb ./fresh --restore ./monday.tessarilog --upto 4200

Into an empty store. --upto stops the replay at a sequence, and that is what point-in-time recovery is here: the log is ordered, so a prefix of it is the store as it was.

--restore tells a snapshot from a log by how the file begins:

$ tessaridb ./fresh --restore ./monday.tessarisnap

A snapshot is restored whole or not at all. A cut log is a prefix of history, which is a state the store once held; a cut snapshot is whatever records happened to come first, which it never did — so one missing its end is refused before anything is written. The restored store stands exactly where the snapshot was taken, which makes the log after it applicable on top:

$ tessaridb ./fresh --restore ./monday.tessarisnap
$ tessaridb ./fresh --restore ./since-monday.tessarilog

The indexes are rebuilt from the records as they are restored, so a restore costs that time; a vector index comes back as REBUILD INDEX would leave it, with its recall unmeasured until you measure it.

A script is restored by running it:

$ tessaridb ./fresh -f ./monday.tessariql

It stops at the first refusal and keeps what ran before it, as any script does. Its header names every part it could not write — a vault's secrets, a queue's holds, a topic's positions, a cluster's topology — and a user comes back with DEFINE USER … PASSHASH, the stored hash, so everyone signs in with the password they always had.

Restoring a script into a live store

RESTORE SCRIPT FROM 'crm.tessariql';

RESTORE reads a script from the node's backup folder — the same folder BACKUP … TO writes, with the same rule for the name — and runs it into the store beside what it already holds. It is how a part taken with BACKUP SCRIPT OF comes back, and it is what the console's Backup tab runs.

It only ever creates. Before anything runs, the script is read against the store, and it may define databases that do not exist yet, the namespaces around them (one that already exists is reused), the analyzers their fields use, and the tables, fields, indexes and records inside what it created. It is refused, with nothing written, when:

• a database it would create already exists — RestoreTargetExists (409);
• it does anything else — a delete, a drop, an alter, a user, a grant, a write into a place it did not create — RestoreRefused, naming the statement by its first words. A whole-store script carries its users, so it is refused here and restores into an empty store with tessaridb <store> -f;
• it runs inside BEGIN … COMMIT, since it commits on its own.

It runs as the caller, and each statement is checked again as the caller's own; the statement itself needs a store-wide owner. If the filling is refused, the places it created are dropped again, so a failed restore leaves nothing behind. The answer names the file, how many statements ran and the databases it created: { path, statements, databases }.

What a backup does not carry

A backup carries what the log carries, and the log deliberately does not carry the two settings that describe this machine — its roles and its endpoints. A restored copy that inherited writable from the node it came from would accept writes it was supposed to forward, and peers told to reach it at the original's address would reach the original.

So a restored node is configured on itself:

DEFINE NODE ROLES serving ENDPOINTS 'newhost:9000';

What a backup does carry is the topology: DEFINE REPLICA is a catalog record and replicates like any other. Which half goes where is the subject of Adding a node, and neither test is the criterion on its own — a backup must carry the peers and must not carry the identity.

A pruned log cannot be backed up — take a snapshot

A log backup is the log from its first record, and a prune removes the first records. So once a node has pruned anything, BACKUP LOG and --backup … --from 1 are refused, naming where the log now begins — and --backup leaves no file behind:

$ tessaridb ./data --backup ./whole.tessarilog --from 1
tessaridb: … sequence 1 is below the start of this log, which begins at 56; …
a pruned store is backed up whole with --snapshot

A snapshot carries the state rather than the history, so it is whole on a pruned store — which is why it is the default. What a prune costs is the history below the horizon — reading an earlier version, or INFO FOR HISTORY OF reaching further back — on the store and on anything restored from it.

A serving node keeps the newest 100 000 records of each log by default — see Bounding the log for another number, or none.

Sealed backups

A node given an encryption key seals every backup it produces, by every route on this page, and a sealed backup opens only where its key is — see Encryption at rest. A plain backup still restores into an encrypted store, and --backup-key-file opens one sealed under another key, which is how a store moves to a new key.

The one thing that has no recovery

There is no way back into a closed store from outside. A lost owner password is a restore from a backup taken while you still had it — which is an argument for taking one now, rather than an argument for a recovery mode. A recovery mode is a back door, and a back door is available to everybody.