Vaults

UNSEAL VAULT WITH 'the operator passphrase';

DEFINE VAULT team;
DEFINE FIELD login ON team TYPE string;
DEFINE FIELD token ON team TYPE string SECRET;

CREATE team:'github' = { login: 'boog', token: 'the secret' };
REVEAL token FROM team:'github';

A vault is a table whose fields marked SECRET are encrypted before the record is written. The ciphertext is the stored value, which is the whole design: the sealing happens below the layer that serves reads, so the search index, the change feed, the replication log and every backup carry ciphertext without any of them having to know what a vault is.

Unlike the other declared stores, it is not a desugaring. There is no combination of DEFINE TABLE and clauses that produces one, because what makes a vault a vault is a key — minted when the vault is declared, wrapped under the store's master key. That is why it is the one declaration that needs the store unsealed: a vault whose key was left for later would refuse every write while INFO reported it ready. (A vault declared with a passphrase of its own wraps its key under that passphrase instead, and needs nothing unsealed — see below.)

The claim, stated exactly

The honest sentence is narrower than "the database cannot read it", and it is worth reading slowly because the difference is the whole of the threat model:

The stored bytes are never plaintext, no key that opens them is written to disk unwrapped, and a sealed or restarted node cannot open anything for anybody.

A running, unsealed server can decrypt — it has to, in order to answer REVEAL. What a vault takes away is every copy of the data that is not that running process.

A vault is strict, and cannot be talked out of it

Every other declared store is schemaless unless you say otherwise. A vault refuses a field nobody declared, and refuses to be made schemaless afterwards.

CREATE team:'gitlab' = { login: 'boog', recovery: 'hunter2' };

Refused — UndeclaredField: recovery is not declared on team. Making the vault schemaless afterwards is refused as well:

ALTER TABLE team SET SCHEMALESS;

Refused — VaultIsStrict: a field nobody declared is a field nothing seals.

The reason is mechanical rather than stylistic. What seals a field is the SECRET marker on its declaration, so a field nobody declared is a field nothing seals — it would be accepted and written in the clear, beside the sealed ones, in the store whose whole promise is that it holds nothing readable.

Unsealing

UNSEAL VAULT WITH 'the operator passphrase';
SEAL VAULT;

The master key exists in memory between those two statements and nowhere else. It is never written to disk unwrapped, never replicated, and never included in a backup — so a second node holding every byte of this one's log holds nothing that opens a secret.

It is per process. A restart seals the store: a node that comes back cannot open anything, for anybody, until a passphrase is presented again. That makes an unattended restart impossible, which is a real operational cost and is stated here rather than discovered during one.

An unseal lasts ten minutes, and then the store seals itself. The period is measured from the unseal and is not renewed by use, so a busy node still closes. A node started with --unseal-for 1h, or with TESSARIDB_UNSEAL_FOR=1h in its environment, keeps its unseals for an hour instead; the flag wins over the variable, and a period of zero is refused at start. To extend an open window, seal and unseal again — unsealing a store that is already unsealed is refused.

The passphrase arrives in the statement, as a quoted string — not an expression, not a parameter, not a name. An expression would put a secret through the evaluator, where it could be concatenated into a message or returned by the very statement that read it. A client need not write it into a statement at all: see without a statement below.

On a store that has never held a vault the first UNSEAL creates the root, and answers initialised rather than unsealed when it does. A mistyped passphrase on the first unseal is the passphrase until somebody who knows it changes it.

Wrong passphrases are throttled like wrong passwords: after three in a row a further attempt is made to wait, whether or not it is right, and is refused as PassphraseThrottled.

INFO FOR SEAL;

Answers whether this process can open secrets and until when: { state, seals_at, unseal_for }, where state is uninitialised (no root yet), sealed or unsealed, and seals_at is the instant an unsealed store closes. It is this process's answer — on a cluster each node answers for itself — and any signed-in user may ask.

CHANGE VAULT PASSPHRASE FROM 'the operator passphrase' TO 'a new one';

Wraps the same master key under a new passphrase. No secret is re-encrypted and the store stays sealed or unsealed as it was; afterwards the old passphrase unseals nothing. The current one is checked under the same throttle, and a store that has never been unsealed is refused with NoVaultRoot. A backup taken before the change still opens with the old passphrase, because the root record travels in the log like the rest of the catalog.

A vault with its own passphrase

DEFINE VAULT team PASSPHRASE 'the team passphrase';
SEAL VAULT team;
UNSEAL VAULT team WITH 'the team passphrase';
CHANGE VAULT team PASSPHRASE FROM 'the team passphrase' TO 'a new one';
INFO FOR SEAL OF team;

A vault declared with PASSPHRASE keeps its key under a key derived from that passphrase instead of under the store's master key. The store's passphrase does not open it, holding every authority in the store does not open it, and it can be declared on a store that has never been unsealed. Declaring it leaves it unsealed for one period, as the store's first unseal does. A vault declared without PASSPHRASE opens with the store's passphrase, and which of the two a vault is cannot change after it is declared. INFO FOR VAULT team says which: custody is 'own' or 'store'.

This is how each team gets a passphrase of its own while shared secrets stay in a vault that opens with the store's.

The named statements act on that vault alone. Its unseal lasts the node's period and is throttled on its own; sealing it leaves the store and every other vault as they were; changing its passphrase re-wraps the same key, so no secret is re-encrypted and a backup taken before the change still opens with the old one. INFO FOR SEAL OF team answers { state, seals_at, unseal_for, custody } — for a vault in the store's custody, the store's own state. Naming a vault that opens with the store's passphrase in UNSEAL, SEAL or CHANGE is refused as VaultUsesStorePassphrase, because unsealing the store opens every vault in its custody and the caller named one.

Anybody who may read in the vault's database may unseal or seal it — the passphrase is what they must also hold — and changing its passphrase needs manage there, as declaring it did. What an own passphrase does not change: a running node holding the unsealed key can decrypt, an administrator can still DROP VAULT it (which destroys the key rather than disclosing anything), and the names and number of its records stay visible as for every vault.

Without a statement

A passphrase written into a statement is a passphrase in whatever keeps statements — a console's history, a client's log line on failure. So the four acts have a surface of their own that carries it as a field:

The wire protocol has a frame for the same four acts (protocol 1.2), whose target is the store or one vault, and every client exposes them as functions — see Clients. The console's Vault tab uses these routes. No refusal and no log line quotes the passphrase.

Writing a record

A vault record is written like any other — field by field, or whole:

UPDATE team:'github' SET token = 'the rotated secret';
UPDATE team:'github' MERGE { token: 'the rotated secret' };
UPDATE team:'github' = { login: 'boog', token: 'the rotated secret' };

The forms differ in what happens to the recipients, and that is the reason to prefer the edit. A field-by-field edit seals only the fields it names, leaves every other envelope untouched, and reuses the record's own data key — so every wrap made by ADD RECIPIENT still opens. Replacing the record whole mints a fresh data key, and the recipient entries were wrapped under the key it replaces, so they are cleared.

One thing is refused, and it is narrow:

UPDATE team:'github' SET token = token;   -- refused
UPDATE team:'github' SET login = login;   -- refused, for the same reason

Refused — VaultEditComputesFromTheRecord.

An assignment on a vault may not read the record. The fields an edit names are the fields you supplied, so writing them needs nothing opened — but an expression that reads the record would have to open a sealed value to answer, and opening a secret is REVEAL, which records itself before it answers. An UPDATE that quietly opened three secrets to compute with them would put plaintext in the server with nothing anywhere saying it had been there.

The refusal does not inspect which field you read, so it covers login as well as token. That is deliberate: the alternative is a rule that works out which references are safe, and a rule like that is wrong on the case nobody thought of, in the direction that returns a secret.

Listing a vault

INFO FOR VAULT team RECORDS;
INFO FOR VAULT team RECORDS AFTER team:'github' LIMIT 100;

The vault's record ids in key order, a page at a time: { records, next }, with next naming the last id when the page was full and NONE when it was the last page. A page is a thousand ids unless LIMIT says otherwise, and at most ten thousand. No value of any record is in it, sealed or not; the ids themselves are not secret, because a record's identity is its key and keys are not encrypted — which is why a vault record should never be named after what it holds. Reading the list needs a grant on the vault, as reading anything in it does.

INFO FOR DATABASE names the database's vaults under vaults, beside tables and topics, so a caller can tell which tables are read this way.

Reading a secret

REVEAL token FROM team:'github';
REVEAL * FROM team:'github';

REVEAL is the only statement that turns a sealed value back into plaintext. It names one record, takes no WHERE and no ORDER BY, and answers with the secret fields it was asked for — * meaning every sealed field, never the ordinary ones beside them.

The absent clauses are the feature rather than a simplification. A filter over a secret is an oracle that answers one bit per statement, and an ordering is the same oracle more slowly.

A field that is not declared SECRET is refused rather than returned in the clear, because a caller reading REVEAL's answer has no way to tell which entries were ever sealed.

What a vault refuses

Sharing: the recipient set

ADD RECIPIENT 'ada@example.com' TO team:'github' KEY 0xdeadbeef;
REMOVE RECIPIENT 'ada@example.com' FROM team:'github';
INFO FOR RECIPIENTS OF team:'github';

The store interprets neither half. The name is text it keeps and hands back; the material is a value it keeps and hands back. What a recipient's material is — a data key wrapped under somebody's public key, a handle into your own key service, a capability your application issued — is a question this store deliberately cannot answer, because answering it would mean holding the second key hierarchy that decides it. The set is the foundation; the sharing scheme is yours.

Four properties worth knowing before you rely on it:

• Neither statement needs an unsealed store. Nothing here unwraps a key, and revoking is the operation you least want to depend on an operator being present to perform.
• Adding or removing a recipient changes nothing else. Every sealed field keeps the exact bytes it had.
• A name already in the set is refused, not replaced. Overwriting would destroy the only copy of whatever that entry held, silently, in one statement.
• Removing a name that is not there is refused, not answered ok. A revocation that matches nothing and reports success leaves you believing a party was removed while their entry is still on the record.

Reading the set needs the same grant as reading the vault. Holding a grant and holding a key remain separate powers — the set says who could open a record, not who may address it.

Every read is recorded, or refused

REVEAL writes a record of itself before its answer leaves, and a read that cannot be recorded is refused rather than served. Both halves matter, and neither is visible when things are working:

• Recorded afterwards, any crash or partial write between the decryption and the log produces a secret release with no trace — and the two orderings look identical whenever nothing fails.
• Served when the trail is broken, the store's accountability becomes the thing an attacker switches off first. The cost of the other choice is real: a broken trail is an outage of every REVEAL.

The record carries who asked, which vault, which record, which field names, and whether it was served — never a value, a fragment of one, or a length that discloses one. Refusals are recorded too, because a denial is the first sign of somebody probing what exists.

INFO FOR AUDIT;
INFO FOR AUDIT BY 'ada';

The second narrows to one actor, which is the shape the question is actually asked in — this credential was compromised; what did it open, and what has to be rotated now?

It is answered only to a caller who administers the whole store. The reason is where the trail lives rather than how sensitive it is: a read is recorded before anybody knows whose tenancy it belonged to, so the trail is held store-wide and there is no tenancy to scope an answer by.

Three limits, stated rather than discovered later:

• The built-in trail lives in this store, so it shares a failure domain and an access path with the thing it audits. An independent device is what a deployment adds; the engine exposes the seam, and every installed device must also succeed for a read to be served.
• There is no tamper-evident chain. Chaining would serialise every read through one key, and what it buys is evidence against an attacker who already holds the backend — which is outside what a vault defends against anyway.
• The trail is scanned, not indexed. INFO FOR AUDIT reads it end to end. That is the honest shape at this size and stops being one long before a busy store's trail does.

What a vault does not defend against

A boundary nobody states is a boundary everybody assumes is covered, so here is this one. A vault defends the stored bytes: an attacker who obtains the whole storage backend, a backup file, a snapshot or a replica gains no secret material, a sealed or restarted node opens nothing, and a ciphertext moved to another field or record fails its binding.

Eight things it does not defend against:

1. The memory of a running, unsealed node. The server decrypts, so while it is unsealed the key and the plaintext are in its address space.

2. Privileged code on the host. Anything that can read that process can read what it holds.

3. A malicious operator — the party who performs the unseal is the party who can open what the unsealing opens.

4. A client already holding a valid credential for a vault it may reach. The store cannot distinguish that client from the person it belongs to.

5. Existence, size, count and access patterns. Record identities are keys and keys are not encrypted, so a vault holding one record named aws-root has said something without a byte being decrypted.

6. Timing.

7. Issuing secrets. A vault holds material you put there. It does not mint short-lived credentials, sign certificates, or encrypt on your behalf.

8. An attacker who can write to the storage backend. Note the difference from the first sentence above, which is about one who can read it: writing is the stronger power and it buys a rollback. Restore one record's stored bytes from an older copy and the secret that was rotated out is live again, because the envelope binds the table, the record and the field — everything that identifies which value this is, and nothing that identifies when.

   A version bound into the envelope would not close it, and that is worth saying because it is the fix that suggests itself: any version the opening side can read from the record is restored along with the ciphertext it was meant to police, so both sides of the comparison move together. Closing it needs a counter the attacker cannot roll back, which means state outside the store — and an embedded engine whose only durable state is the backend under attack has none to offer.

   What follows operationally is short. Rotate at the source, not only in the vault — a credential the issuing system has revoked is dead whichever copy of the ciphertext comes back — and treat backend write access as equivalent to holding the secrets.

Removing a vault

DROP VAULT team;

This destroys the vault's key along with its records. In this store the key is gone permanently, and every copy taken from this point on is ciphertext under a key that exists nowhere. That is what makes it a deletion rather than a removal, and it is the only deletion claim a store can honestly make: deleting rows is a statement about the live table and says nothing about the data.

Where the deletion stops. A vault's key lives in its declaration, the declaration travels in the log, and a backup is the log — so a backup taken before the drop restores a vault that opens, given the passphrase. This is the same mechanism that lets a replica serve a vault at all, and it cannot be removed from one side without removing it from the other.

So destroying a secret is two acts, not one: drop the vault, and expire the copies that predate the drop. A retention policy that keeps ninety days of nightly backups keeps ninety days of the secret you just destroyed. The store cannot do the second act for you — it does not know what copies exist — and a claim that it did would be the most dangerous sentence on this page.