Queues

A queue is a table you declare with a deadline. CLAIM hands you a record and writes the moment its hold expires; if you do nothing until then, the record becomes claimable again on its own. That is the whole mechanism, and it is worth stating plainly because most of what a queue usually needs — a lease manager, a heartbeat, a reaper job — is absent here, not missing.

Declaring one

DEFINE QUEUE jobs TIMEOUT 30s;

TIMEOUT is required and has no default, for the same reason DEFINE VECTOR's width is: declaring it is the whole capability. A queue whose holds never lapse is an ordinary table with two extra fields, and a timeout the store picked for you would hand work to a second worker at a moment nobody chose.

A queue is not a shorthand. DEFINE VECTOR and DEFINE GEO stand for a collection, a field and an index you could write by hand; there is no such spelling for a queue, because a claimed_until field on a plain table is a field, not a hold. Nothing enforces it, nothing lapses, and nothing passes over it. What makes a queue a queue is the kind, which is why the kind is what the store keeps — and why the same two field names stay perfectly ordinary on a table that is not one:

DEFINE COLLECTION notes;
CREATE notes:2 = { attempts: 3, claimed_until: 'whenever' };

That is accepted. On a queue, both fields belong to the engine and are refused to you on the way in:

CREATE jobs:9 = { url: 'd', claimed_until: 1 };

Refused — QueueFieldIsTheEngines, and the same refusal on an UPDATE, because a field you cannot set at creation but can set afterwards is not protected.

Putting work in, and taking it out

Records arrive the ordinary way — a queue is a table, so CREATE is CREATE:

CREATE jobs:1 = { url: 'a' };
CREATE jobs:2 = { url: 'b' };
CREATE jobs:3 = { url: 'c' };

CLAIM FROM takes the head of the queue and holds it:

CLAIM FROM jobs;

Order is by identity, which for the counter identity is the order the records were written. The next claim passes over what the first is holding and answers with a different record. When nothing is claimable, the answer is no records and no error — an empty queue is a normal state of a queue, not a fault:

CLAIM FROM jobs;   -- rows 0, once everything is held

Ask for several at once, and you get at most that many:

CLAIM 5 FROM jobs;

There is a ceiling on how many one statement may take, and asking above it is refused (ClaimAboveCeiling) rather than quietly trimmed. CLAIM 0 is not a claim at all, and the grammar says so.

Handing a record back

RELEASE jobs:1;

The hold ends immediately and the record is claimable again, without waiting for the deadline. This is what a worker does when it finishes — or when it finds it cannot do the work after all.

On a queue that declares NOT BEFORE f, a worker that wants to try again later says when, in the same statement (from 0.33.4-beta):

DEFINE QUEUE later TIMEOUT 30s NOT BEFORE due;
CREATE later:1 = { job: 'resize' };
CLAIM FROM later;
RELEASE later:1 NOT BEFORE 10m;
CLAIM FROM later;

RELEASE … NOT BEFORE takes a datetime or a duration from now, sets due and ends the hold in one write, so no worker can claim the record between the two. The second claim answers no records. RELEASE ALL FROM later NOT BEFORE 10m does the same for every hold the session has. A queue that declares no NOT BEFORE field refuses it as NoDelayField.

The hold that lapses

The deadline is the recovery mechanism, and there is nothing else. A worker that crashes, loses its network, or is killed mid-task never releases its record; the deadline passes, and the next CLAIM hands the record out again. No process notices the death, because none has to.

DEFINE QUEUE fast TIMEOUT 1ns ATTEMPTS 2;

With a timeout that has already elapsed by the time the next statement runs, the same record is handed out twice in a row — which is the expiry, watched directly.

The attempt ceiling, and the dead letter that is a predicate

DEFINE QUEUE retries TIMEOUT 30s ATTEMPTS 5;

Every hand-out counts an attempt, whatever the worker did with it. Once a record reaches the ceiling it is no longer handed out, and the next claim moves on to the record behind it.

ATTEMPTS is optional, and leaving it out means unlimited. That is a legitimate choice for work that cannot poison, and a visible one: the absence of the clause is what says it.

What it does not do is move the record anywhere. The dead letter here is a predicate, not a second table:

SELECT url FROM fast WHERE attempts >= 2;

This is deliberate. A dead-letter table is a copy of your records under a second name, with its own retention, its own permissions and its own opportunity to drift out of step with the original. A query over a field is none of those things, and it composes with everything else the language can filter on.

The order work is handed out in

DEFINE QUEUE alerts TIMEOUT 1m PRIORITY BY severity;
DEFINE INDEX by_severity ON alerts FIELDS severity;
DEFINE QUEUE mail TIMEOUT 5m ATTEMPTS 3 NOT BEFORE send_at;

CREATE alerts = { severity: 2, text: 'disk at 80%' };
CREATE alerts = { severity: 9, text: 'disk full' };
CREATE mail = { to: 'ada@example.test', send_at: time::from_unix(time::unix(time::now()) + 600) };

CLAIM FROM alerts;
CLAIM FROM mail;

Without either clause a claim takes records in arrival order. From 0.22.0-beta two clauses change that.

PRIORITY BY f hands out the greatest value of f first — the severity: 9 alert above, before the one that arrived earlier — in the order ORDER BY f DESC uses, so records sharing a value go in arrival order and a record without f goes after every record that has one. A value index on f makes finding the head a walk down the index; without one the claim reads the queue and keeps the best. The index changes what a claim costs, never which records it takes: the claim reports ordered with the index's name when one served it, and scan when none did.

NOT BEFORE f holds a record back until the instant in f: the mail above is not handed out for ten minutes, so the second claim answers no records. It is delayed delivery written as a value in the record, the way a hold's deadline is — every node compares the same instant, nothing sweeps, and a worker that wants to retry later updates the field when it releases. A value in f that is not a datetime is no delay at all, so a mistyped value cannot keep a record waiting forever with nothing in an error state.

Work on a schedule

There is no schedule statement. A tick is one record whose NOT BEFORE field is its next run:

DEFINE QUEUE ticks TIMEOUT 5m NOT BEFORE next;
CREATE ticks:1 = { job: 'cleanup', next: time::now() };
CLAIM FROM ticks;
RELEASE ticks:1 NOT BEFORE 1h;

A claim is a write fenced by leadership, so two workers on two nodes cannot both hold the tick. A worker that dies holding it loses it when the hold lapses, and another takes it: a run can happen twice across a leader change, but it is never skipped. A tick missed while no worker ran is claimable once. Re-arming from time::now() coalesces the backlog into that one run.

Enqueueing once

DEFINE QUEUE reclaim TIMEOUT 1m DEDUPLICATE 10m;
CREATE reclaim:'bucket/key' = { reason: 'overwritten' };
CREATE reclaim:'bucket/key' = { reason: 'overwritten' };
SELECT * FROM reclaim;

DEDUPLICATE d makes a CREATE of an identity first written less than d ago write nothing and answer no record — whether that identity is still queued, held by a worker, or done and deleted (from 0.33.4-beta). Without the clause a queued identity is refused as RecordExists and a finished one is queued again. The window counts from the first write and a repeat does not stretch it. A record the store names has no identity to repeat and is never deduplicated. INFO FOR TABLE reports deduplicate.

Claiming the record you name

Sometimes the caller already knows which record it wants — it is retrying its own work, or acting on a request that named one. CLAIM <record> takes it by name:

CLAIM jobs:1;

This is the same write as the selecting form, on the record you name, and the walk then passes over it exactly as it would any other hold. It differs from CLAIM FROM in what it does when it cannot give you the record, and the two answers are different on purpose:

Contention is an answer — somebody else got there first, which is the normal outcome of a race and not a fault. Existence is an error, because a caller who named a record it believes exists has a bug worth being told about, not an empty result to misread as already taken.

Because the targeted form skips the walk, arrival order is a promise of CLAIM FROM alone.

RELEASE ends a targeted hold the same way it ends any other — and it now asks who holds the record, which it did not use to. See Saying who you are below.

Saying who you are

USE CONSUMER 'billing';
CLAIM FROM jobs;
RELEASE jobs:7 FOR CONSUMER 'billing';
RELEASE ALL FROM jobs;
RELEASE ALL FROM jobs FOR CONSUMER 'billing';

A session that says who it is signs the holds it takes, and the record carries claimed_by — an object of two routes, consumer and instance. A session that says nothing signs nothing, and its holds stay releasable by anybody, which is exactly how a queue behaved before this existed.

The name is yours and it is meant to be shared. Several workers under one name is the ordinary arrangement, and it means we are the billing workers — they divide the work between them and none displaces another. Only the instance is unique, and you never write it: the store mints one for each session that declares a name, so two workers cannot collide however they are configured. There is nothing to fence and nothing to configure.

RELEASE ALL FROM jobs reaches this session's own holds, and answers the records it freed rather than a count — a session coming back from a crash is the caller least able to read anything first. That is the safe form, and it is what a worker uses when it is shutting down.

The FOR CONSUMER forms reach the whole group, live sessions included. A restarted worker is a new instance, so reclaiming what its predecessor left is the named form. RELEASE ALL FROM jobs FOR CONSUMER 'billing' frees everything that group holds in that queue; RELEASE jobs:7 FOR CONSUMER 'billing' frees one record. Both are spelled out precisely because they can take work from somebody still doing it — and a client that holds one connection for many logical callers needs the second one, because it is minted a fresh instance every time it declares a name.

A hold nobody signed belongs to no group, so the named forms leave it exactly where it is. The bare RELEASE still frees it.

Releasing a hold that belongs to another consumer is refused, naming the consumer that holds it. The store protects you from a mistake, not from a lie: a consumer name is a string the client chose, and a caller who may write the table can choose any of them.

RELEASE ALL names its queue for the reason CLAIM names its queue. A form that named none would sweep every queue in the database, and a caller granted write on only some of them would get a partial success that read as a whole one.

A queue says whether it is strict and which graph it is in

DEFINE QUEUE jobs  TIMEOUT 30s SCHEMAFULL;
DEFINE QUEUE tasks TIMEOUT 10m SCHEMAFULL IN work;
DEFINE FIELD url ON jobs TYPE string REQUIRED;

Both words mean here exactly what they mean on an ordinary table. They come after TIMEOUT and ATTEMPTS — those are what a queue is, and these are adjectives on it — they are order-free against each other, and neither is accepted twice.

Leaving both out means what it has always meant: lenient, and in no graph. That is not a special case for queues. It is the rule any declaration carrying no columns already followed, since strictness constrains declared fields and a queue declares none until a DEFINE FIELD arrives afterwards.

The clauses exist because a work table usually wants both. A table that holds work is normally the one a schema is strictest about, and it is normally an end of a link — a task belongs to a goal, a job blocks another job. Before these words such a table could not be a queue at all, and not in a way you could work around: DEFINE EDGE refuses a table belonging to no graph, declaring the table first and the queue second is refused because the name is taken, and ALTER TABLE … SET SCHEMAFULL does reach strictness while leaving a table INFO FOR TABLE can no longer write back as a statement. Say it in the declaration.

The three fields the engine writes onto a record it hands out — attempts, claimed_until and claimed_by — are declared by nobody, and they are written after your record has been validated. They are excused by the table's kind, so a strict queue does not refuse the claims it has just taken; and a caller who writes them is still refused, because they are the engine's.

A write that says nothing about those three carries them forward. A whole-record UPDATE jobs:7 = { url: 'b' } names none of them, and a whole-record write otherwise removes every field it does not name — so without this rule a worker rewriting the record it holds would get it back unheld, with its attempt count cleared and nothing anywhere in an error state. A caller may not introduce or change these fields, and it follows that a caller may not remove one either.

Dropping one

DROP QUEUE fast;

The table and its records go together, held ones included. DROP QUEUE refuses a table of another kind rather than dropping it, for the same reason DROP VECTOR does: the two words name different things even where they would remove the same rows, and a typo should not have the blast radius of a table.

What a claim is, underneath

A claim is an ordinary write inside the ordinary transaction. It goes through the same batch as every other change, which means it replicates like one, recovers like one, and is visible to the rest of the transaction that made it. A second claim inside one BEGIN sees the hold the first one wrote.

There is no lease service, no ZooKeeper-shaped dependency and no clock agreement between workers to arrange, because there are no workers in the protocol — only records with deadlines, in a store that already knows how to write records atomically.

What it is not

A queue is not a message broker. There is no fan-out to several consumers, no topic, no acknowledgement protocol on the wire, and no order beyond arrival and the PRIORITY BY a queue declares. What it is good at is the case most applications actually have: work items that must be handed to exactly one worker at a time, that must come back if that worker dies, and that must not be retried forever.

For reading an external stream into the store, see Stream ingestion, which is a different job.