Delivery

Components interact by messages: a call and its reply, an event, a task. In a distributed system a message can be delayed, its recipient can be absent, and the work it triggers can fail halfway. The runtime has an answer for each of those cases, and one obligation follows for application code.

§Distributed exception handling

Inside one process, a failure has an obvious path. An exception unwinds the stack until something catches it, and whoever asked for the work finds out that it failed.

Across components there is no stack. The operation that fails may run in another process, on another machine, or minutes after the request that led to it was answered. Without a common answer, each service handles this in its own way: a retry wrapper here, a dead-letter queue there, a convention for error payloads that the next team does not share. Failures then get lost at the seams between those conventions.

Toa treats this as the platform’s problem and gives it one rule: a failure reaches whoever is waiting for the outcome, or, where nobody is, it is tried again and then kept. No failure disappears, and no component implements the handling.

§Two situations

How a failure is handled depends on a single question: is anybody waiting for the outcome?

Somebody is waiting. A call made with context.remote suspends its caller until the reply arrives. Whatever happens, the caller will be told, and the caller decides what to do next.

Nobody is waiting. An event has no caller: the component that published it moved on at commit. A task is a call whose caller chose not to wait. If the work fails here, there is nobody to tell, so the runtime itself is responsible for seeing it through.

§When somebody is waiting

A call is addressed to a component, not to a process. If no replica of that component is running — it is restarting, being deployed, or scaled down — the call waits for one to return. The caller observes a slow call rather than an error.

Once the operation runs, the caller receives exactly one of the outcomes described in Replies: an output, an error the operation returned, or an exception thrown at the point of the call.

A call whose process stopped before answering is handed to another replica, so a crash is absorbed rather than reported. An exception thrown by the operation is different: the runtime does not run the operation again on the caller’s behalf. Only the waiting party can judge whether asking again is right. In a chain of calls, an uncaught exception propagates back to the origin: the operation that received it fails in turn, until the failure reaches whoever started the chain.

§When nobody is waiting

The two kinds of outcome diverge sharply here.

An error ends the work. A returned error is an answer, and the operation gave it. The message is considered handled. If the refusal has to be visible to someone, the operation makes it visible itself: by changing state, which may in turn produce an event.

An exception has the work tried again. The runtime redelivers the message after a pause that grows with each attempt. Transient conditions — a dependency that was briefly unavailable, a conflict that resolves — clear on their own this way.

A message that keeps failing is kept. After its attempts are spent, the message is set aside, together with what it was and why it stopped, where an operator can inspect it and put it back. It is not dropped and it does not block the messages behind it.

A failure that cannot pass is kept at once. A request that does not fit the operation’s contract will not fit it on the next attempt either. The runtime recognizes failures of this kind and sets the message aside without retrying.

The same holds for a message whose recipient is absent: an event or a task waits for its component for as long as it takes.

§At least once

Taken together, these rules give delivery a precise meaning: a message is processed at least once. It will not be lost. It may be processed more than once. This is most visible where nobody is waiting, because the runtime retries there, but a call somebody waits for can be delivered twice as well.

Repetition has several sources, and none of them can be eliminated:

• an operation failed after doing part of its work, and is retried from the beginning;
• a process stopped after finishing the work but before acknowledging the message;
• an acknowledgement was lost, and the message was delivered again.

From the outside these cases are indistinguishable, so the runtime resolves all of them the same way: by delivering again.

§What this asks of an operation

An operation that changes state has to tolerate running twice for the same message, and one that events or tasks can reach will certainly meet the case. The simplest way is to write the change so that repeating it leaves the same state:

// repeated, charges twice
wallet.balance -= input.amount

// repeated, ends the same
order.status = 'paid'

Where the change is inherently cumulative, the message itself says whether it was seen before. Every event carries the VERSION of the entity it is about, which only grows:

export async function transition (input: Input, entry: State) {
  if (entry.applied >= input.VERSION)
    return

  entry.applied = input.VERSION
  entry.total += input.amount
}

For changes to a component’s own state, the runtime can also take this responsibility entirely: see Transactional Inbox.

What an operation does outside its state — an email, a payment through a third party — is repeated in full on every attempt. The runtime cannot undo or detect such an action. Continuity addresses that case.

§Summary