Clusters

Setting TESSARIDB_S3_ERASURE makes a node a cluster member. Without it the node stores whole objects on its own drive.

Erasure coding

A cluster writes objects with a Reed-Solomon code k+m (k, m ≥ 1, k + m ≤ 16): each object becomes k data shards and m parity shards, one per node of an erasure set. Objects are placed on the sets by TESSARIDB_S3_CLUSTER_SECRET, which is the same on every node, so every node computes the same placement without asking anyone.

• A write is acknowledged on its quorum: k shards durable, or k + 1 when k = m, so that two halves of a split cluster cannot both accept a write.
• A read takes any k shards that verify. A missing or damaged shard is rebuilt from parity for that read and the object is listed for healing.

Failure domains

TESSARIDB_S3_ZONE names the failure domain a node is in: a zone, a rack, a room. A layout never puts more of one object's shards in one zone than its parity can lose, so losing a whole zone leaves every object readable. The first layout takes the registered nodes in id order, passing over a node whose zone already holds m shards.

Healing

Every member runs a healing pass (TESSARIDB_S3_HEAL_INTERVAL_SECS, a minute by default) that rewrites the missing or damaged shards of objects listed for healing, one node at a time per object, under a claim in TessariDB. Shard bytes no object can reach any more are swept.

Changing the layout

A layout holds one or more erasure sets. Root changes it in the console: a layout is staged over the current version, judged against the registered nodes and their zones, and applied as the next version. Either step is refused if the layout or the staging changed meanwhile, needs a fresh sign-in and a reason, and is recorded.

Once a layout is applied, each member moves objects written under an older layout onto it: it copies and verifies the shards whose node changes before switching the object, and removes the old shards once the reclaim grace has passed, at most TESSARIDB_S3_MOVE_BYTES_PER_SEC (32 MiB a second by default; 0 pauses moving). The console's cluster view counts the objects still to move.

Membership

Members are admitted, not discovered:

• A cluster begins from its founding members (TESSARIDB_S3_FOUNDING_MEMBERS).
• Any other node joins only when root approves it in the console, or with a one-time join token (TESSARIDB_S3_JOIN_TOKEN) issued in the console, which binds the node's certificate. A node that is not an admitted member does not start.
• Root removes nodes for good; a removed node is never admitted again, and the removal reaches every node within 5 seconds.

The internal shard surface speaks mutual TLS only (TLS 1.2 and 1.3). Each node's certificate is issued for <node id>.peer.tessari-s3; a node whose certificate names another node does not start, and a call reaches another node only when its certificate is issued for the node dialled and that node is an admitted member.

Regions

A server's region (TESSARIDB_S3_REGION) is the one requests must be signed for. Its buckets' objects are kept in the metadata database <database>_<region>, and a request for another region's bucket is answered 301 PermanentRedirect.

A bucket's replication rules (PutBucketReplication) copy each version a rule covers to a bucket in another region, keeping its version id; the version answers x-amz-replication-status, and a copy that fails is retried after a growing wait and marked FAILED after 10 tries. From the console, root fails a region over to another and restores it, each under the next epoch; while failed over, the region refuses client writes with ServiceUnavailable and still takes replicas.

Measured

Measured on 0.0.1-alpha: six processes as one 4 + 2 cluster on a single laptop, the nodes speaking mutual TLS to each other, all six data directories on one SSD, metadata in a TessariDB 0.34.2-beta node, release build, warm cache, median of three runs, every GET returning exactly the bytes PUT:

On that machine writes are limited by the drive: every PUT syncs its six shard files, and on macOS each sync flushes the whole drive's cache. On an earlier build, with one node of the six stopped every request still succeeded; reads took about twice as long, because a missing shard is rebuilt from parity on every read.