TessariDBDocs
  • 0.34.2-betalatest
  • 0.34.1-beta
  • 0.34.0-beta
  • 0.33.4-beta
  • 0.33.3-beta
  • 0.33.2-beta
  • 0.33.1-beta
  • 0.33.0-beta
  • 0.32.0-beta

Upgrading a cluster

Moving a running cluster to the next minor release one node at a time — followers first, the store's leader last — and the three rules that keep every record.

A cluster running 0.23.0-beta or later moves to the next minor release without stopping. Replace one node at a time: stop it, start the new release on the same store directory, and wait for it to answer before the next. Each replaced node opens the store the previous release wrote and catches up from its leader, and the cluster takes writes throughout — a rolling upgrade.

start step 1 step 2 step 3 n3 · follower n2 · follower n1 · leader old new new new old old new new old old old new turn on what the new release adds only after step 3
Followers first, the store's leader last, one node at a time on its own store. Writes land after every step.

The order

  1. Find the leaders. INFO FOR NODE reports cluster.leaders — which node leads the store line and each placed range — and the console's cluster map draws it.
  2. Replace the followers, one at a time. Stop the process, start the new release on the same store directory with the same flags, and wait until it answers and has caught up before the next.
  3. Replace the store line's leader last. While it is down, writes it would have taken are refused until another node is elected — about one lease and one election, as on any failover. A node leading a placed range pauses that range the same way; to avoid even that pause, move the placement to a node already upgraded first.

What is promised, and what is not

  • One minor version at a time — 0.23 to 0.24. Two steps are two upgrades, each done in full before the next starts.
  • Back is a restart, until you finalize. From 0.31.0-beta, a newer release keeps the on-disk format the store already holds, so the release before can still open it — see taking an upgrade back. Take a snapshot before you start anyway.
  • New features after the last node. While both releases run, use only what both understand. Turn on what the new release adds — BALANCE LEADERSHIPS, LEADS … PREFERRED, giving a range back with LEADS NONE, a transaction across two leaders — once every node runs it. An older node does not know them, and a decision only half the cluster can read is not one it makes.
  • Replication changes shape on its own. From 0.31.2-beta a leader sends its followers each commit as it lands. That happens only between two nodes of that release or later; any other pair streams as before, so a cluster of mixed versions replicates throughout. It needs no switch.
  • Gathered reads fold more as leaders are replaced. From 0.32.0-beta a node holding part of a split table asks a shard's leader to fold median, collect and the counter folds only when that leader runs 0.32.0-beta or later; an older leader sends its records, as before. The answer is the same either way; what travels shrinks once the last leader is replaced.
  • Approximate folds and expiry wait for the last node. From 0.33.0-beta a leader folds approx_distinct and approx_quantile itself only when it runs 0.33.0-beta or later; an older leader sends its records. Declaring that a table's records expire, or a rollup that keeps those folds, waits on an older store for ALTER STORE FINALIZE FORMAT — run it only once every node runs 0.33.0-beta and no rollback to an older build is planned.

Measured on three local processes going from 0.23.0-beta to 0.24.0-beta, and again from 0.31.1-beta to 0.31.2-beta: each node replaced in turn, writes landing after every step, and every node holding every record at the end.

Taking an upgrade back

A store keeps the format it held when the newer release first opened it. The newer release reads and writes it in that format, and refuses — with FormatNotFinalized, writing nothing — the one statement that needs its own format today: a containment index (DEFINE INDEX … CONTAINS). Going back is stopping each node and starting the previous release on the same directory.

INFO FOR STORE shows where a store stands:

INFO FOR STORE;

{ format: 5, namespaces: ['shop'], writes: 6 }

format is what the store holds and writes is what this release writes; the console shows both on This node. When they differ, the way back is open.

When no older release will open the store again, close it:

ALTER STORE FINALIZE FORMAT;

It raises the store to the format this release writes, on every replica, and answers { format: 6 }. It is one-way: afterwards the previous release refuses to open the store, naming the format it found, and the way back is the snapshot taken before. On a cluster it is refused — FormatPeerTooOld, naming the replica — until every declared replica has collected from this node running a release that writes the format. Upgrade every node, let each collect once, then finalize. It needs store-wide authority.

A store a release creates holds that release's format from the start, so a store created by 0.28.0-beta or later already holds format 6 and sees no difference.

Measured on the store 0.27.1-beta wrote: 0.31.0-beta opened it at format 5 and refused a containment index, 0.27.2-beta opened it again and read back every record including the ones 0.31.0-beta wrote; after the finalize, 0.27.2-beta refused it.

From an older release

The first step this covers is 0.23 to 0.24. A store written by an earlier release is moved by taking a script backup and restoring it into the new release as a fresh cluster.

Upgrading a cluster — TessariDB