Incidents

The emergency operator path

tessaridb-oauth bootstrap-operator --email root@example.com
tessaridb-oauth bootstrap-operator --company acme --email ops@acme.example

For the first operator after installation, or one whose access is lost. It reads the server's environment and needs only the store. Without --company it works in the system company, creating it when there is none, and gives the roles companies and users. With --company the company must exist and be served (an unknown, provisioning or suspended one is refused and nothing is written), and the roles are users, clients, policy, keys and audit. An account with no password gets a one-time enrolment code, printed once and never logged. Every run is logged at ERROR as break_glass and recorded on the company's audit trail.

A signing key that may be exposed

When a company's private signing key may have been read by someone else, withdraw it at once; there is no reason to wait.

1. Find it: GET <ADMIN_URL>/admin/<company>/api/keys lists each key with its kid and state. The one to withdraw is usually the signing one.
2. Withdraw it with an operator holding keys: POST <ADMIN_URL>/admin/<company>/api/keys/<kid>/withdraw with a reason. It leaves the key set in the same request. When no other key could sign, a replacement is made first and signs at once; otherwise a key already published takes over.
3. Expect what the key signed to stop verifying. A service that fetches the key set again refuses those tokens at once; one that cached it may accept them for up to five minutes, the key set's cache lifetime and also how long an access token or ID token lives. Refresh tokens are not signed and need no revocation.
4. Tell the company's integrators which kid was withdrawn and when, so a service that pins keys or caches them longer can drop it.
5. Check the audit trail for the keys.withdraw entry and anything else done with the same access.

Every company's keys are sealed with the same key-encryption key. If that file may be exposed too, withdrawing keys is not enough: re-seal, below.

A key-encryption key that may be exposed

The KEK seals five kinds of value in each company's database: its signing keys, the salt its pairwise subjects are made from, each authenticator app's secret, each authenticator being added, and each queued mail. To move all of them to a new key:

1. Stop every node of the region. A node still serving seals what it writes under the old key.
2. Make the new key: head -c 32 /dev/urandom > new.key && chmod 600 new.key.
3. With the server's environment and TESSARIDB_OAUTH_KEK naming the new file, run tessaridb-oauth reseal --old-kek <old file>. It seals every value again under the new key, then walks them again to check that each opens under it. Its last line counts what it found, for example {"companies":2,"provisioning":0,"resealed":9,"already":0,"unopened":0,"finished":true}, and it exits 0 only when finished is true.
4. Start the nodes with the new key.

A run that stops part way can be run again: a value already under the new key is counted already and not written. unopened counts values that open under neither key; they are left as they were and the run does not finish until they are explained. The command refuses when the old and new files hold the same key.

What re-sealing does not do:

• Backups taken before it still open with the old key. Destroy the old file only once every such backup has expired or been destroyed.
• If a copy of the store or a backup was taken together with the exposed key, the sealed values may have been read. Also withdraw every company's signing keys, have users add their authenticator apps again, and expire every backup sealed under the old key.
• The pairwise salt is kept, because a new one would change every sub a client knows its users by.

A leaked client key

When a client's private key, or the host that holds it, may be in someone else's hands, remove the client with an operator holding clients: POST <ADMIN_URL>/admin/<company>/api/clients/<id>/remove with a reason. The key stops authenticating at once, every grant to the client and every refresh family it held are revoked, and access_tokens_end_by says when the access tokens it holds stop working, at most five minutes on. Register it again with a new key; it may keep its client_id and starts with nothing, so every user is asked again.

A taken account

When an account may be used by someone other than its owner, disable it with an operator holding users: POST <ADMIN_URL>/admin/<company>/api/users/<user>/disable with a reason. Every session ends at once (its clients are told through the back channel), every refresh family is revoked, nobody can sign in to it, and its access tokens stop working by access_tokens_end_by. Once the owner is reached, enable it again and have them ask for a password reset from the sign-in page; the reset keeps the account's second factor.