Cadence

Not all work begins with a request. Expired orders have to be cancelled, reports rebuilt, a reminder sent a week after a signup. In most systems this is the job of a scheduler that lives beside the application and calls into it.

In Toa, time is another source of calls to operations. A component declares the work it does on its own time, and the runtime makes the calls. There are three forms: a pulse, which recurs on an interval; a schedule, which recurs at the moments of a calendar; and a delay, which happens once.

§Pulse

A pulse calls an operation of its own component at a regular interval:

# orders/manifest.toa.yaml
cadence:
  expire: 3600

Every hour, the expire operation of orders is called. Nothing is stored to make this happen. The schedule is a function of the clock, so there is no job table to maintain and nothing to restore after a restart.

A component usually runs as several replicas. A pulse belongs to the component, not to a process: the replicas agree on which of them makes each call, and one does.

§Spreading a cycle

A pulse can split its cycle into intervals:

cadence:
  sweep:
    cycle: 86400
    intervals: 24

The operation is called every hour and is told which of the 24 intervals the call is for. An operation that treats the interval as its share of the work processes a twenty-fourth of the data each hour, instead of all of it at midnight.

§What a pulse promises

A pulse is deliberately simple, and its guarantees follow from having no stored schedule.

An interval may be missed. If nothing was running when an interval came due, or the operation failed, that interval is not made up later.

So the operation selects by state. “Cancel every order that is past its deadline” recovers from a missed interval on the next one. “Cancel the orders that expired in the last hour” loses them. A pulse is the right tool when the state itself says what is still due.

A call does not overlap the previous one. If the work outlasts its interval, the next interval is skipped.

§In one replica, or in every one

A pulse is the component’s by default: its work is over shared state — a collection to sweep, a report — and is done once, however many replicas run.

Some periodic work concerns what one process holds: a cache to trim, a buffer to flush. Such a pulse is declared to run in every replica, and each replica then looks after its own:

cadence:
  trim:
    cycle: 60
    scope: replica

§Schedule

A pulse counts from the epoch, so it knows hours and days and nothing of a calendar. A schedule calls an operation at the moments a calendar names:

# reports/manifest.toa.yaml
cadence:
  weekly:
    schedule: 0 9 * * 1 # nine on Monday morning
    zone: Europe/Berlin # in Berlin

The operation is called with the moment the call was scheduled for, which is not the moment it is made: a report produced twenty minutes late is still the report for nine o’clock.

§What a schedule promises

A schedule is stored, which a pulse is not, and that is the whole difference between what the two promise. Each occurrence is a delayed call, handed over ahead of its moment.

A call is made late rather than lost. A component that is down at nine is called when it is back, with the moment it was owed for.

At most one occurrence is owed. A component that was away from Friday to Monday is called for Monday and not for Friday, and an outage of a month does not return as thirty calls. A schedule may state a tighter bound on how late a call is still worth making.

It may be made more than once, so an operation for which a second call would cost something refuses it by the moment it was scheduled for.

Calls may overlap. A pulse skips an interval its work did not fit in; a schedule does not.

An operation may have several entries of either kind — a pulse for what is due every hour, a schedule for everything on the first of the month — and each is made independently.

§In one region, or in every one

An application that runs in several regions has every region make every pulse and every schedule, each for itself. Work that is the whole application’s — an invoice run, a report — names the one region that makes it:

cadence:
  sweep:
    cycle: 3600
    region: 0 # a pulse only region 0 makes
  invoice:
    schedule: 0 6 1 * *
    region: 0 # a schedule made once, however many regions there are

Delayed calls are kept in every region, and each region makes its own. When a region is lost, another is told to make its calls as well — its delays, and the pulses and schedules declared for it — and what it takes over it keeps.

§Delay

A delay hands over one call to be made later:

const id = await context.delay(
  'messages.remind',
  { input: { customer: order.customer } },
  { interval: 7 * 24 * 3600 * 1000, overdue: null }
)

Like a schedule, a delay is stored. It survives restarts and deployments, and it is made when its time comes even if nothing was running in between. The identifier it returns cancels it.

A delay fits work where each item has its own moment: this invoice tomorrow, that one in a week. A pulse or a schedule fits work that sweeps over everything on a common rhythm.

§How late is too late

Every delay states how late its call may be made and still be worth making. A payment reminder is useful whenever it arrives. An order that expires at noon should expire at noon or not at all.

Only the caller knows which kind of call it is making, so there is no default. The caller chooses between no bound, an exact bound, and “on time or never”.

§How precise a delay is

Calls waiting to be made are looked over on a period, a minute unless the application states another. Everything a delay promises is measured against it: a delay shorter than the period is made at the next pass rather than at its moment, a lateness bound shorter than the period can be missed with nothing wrong, and a cancellation is reliable only while the call is further out than that. An application that needs seconds asks for a period of seconds, and pays for a pass that runs that much more often.

§What a delay promises

The call is made at most a bounded time late, or not at all, as the caller stated.

It may be made more than once. Like any message, a delayed call can be repeated, and the target is where a repetition is refused; see Transactional Inbox.

Arming it is not part of the operation’s commit. A delay is stored separately from the operation’s own state change. An operation that must not lose the call hands it over first.

Arming it is a write. A call that may only read cannot arm a delay, whatever the delayed call itself would do.

One attempt is made to hand it on. Whether the target operation then succeeds is the target’s concern, under the rules for tasks.

§Time and chains

A pulse and a schedule start a new chain with every call, since the call comes from the clock and not from a request. A delayed call continues the chain that armed it. An operation that re-arms itself is therefore a circle, however slow, unless it says that each run is a fresh start.

§What it rests on

A delay, a schedule, and a pulse that is the component’s all need the replicas to agree on who makes a call: they rest on atomicity. Where it is not configured, none of them is made. A pulse that runs in every replica asks for nothing, and is made either way.