Run Work on a Schedule

Two requirements of the shop are about time, not about a request: unpaid orders are cancelled every few minutes, and a sales report is sent every Monday at nine. Both are declared under cadence in the manifest of the component that does the work. A number is a pulse, a regular interval; a string is a schedule, a calendar.

§Every few minutes

Name an operation of the component and state the seconds between calls. The orders are those of store.orders, the component Create a Component writes; add to its manifest:

# components/store.orders/manifest.toa.yaml
cadence:
  expire: 300 # every five minutes

operations:
  # …
  expire:
    type: effect
    input:
      type: object
      properties:
        n: { type: integer }
        i: { type: integer }

// components/store.orders/operations/expire.ts
import type { Context, ExpireInput } from '../types/index.d.ts'

const HOUR = 3600 * 1000

export async function effect (input: ExpireInput, context: Context) {
  const deadline = context.now() - HOUR

  const orders = await context.local.enumerate({
    query: { criteria: `status==pending;CREATED<${deadline}`, limit: 100 }
  })

  for (const order of orders)
    await context.local.cancel({ query: { id: order.id }, input: { reason: 'unpaid' } })
}

expire is an effect. It lists the orders with the inherited enumerate and cancels each with cancel, the transition Announce a Change writes, which takes a reason. context.now() is the current time in milliseconds.

Every five minutes one replica of store.orders calls expire, however many replicas run.

The operation a pulse or a schedule names:

• is an operation of the same component; another name is refused when the manifest is read: Cadence refers to undefined operation 'expire';
• is a transition, assignment, effect or unmanaged; an observation or a computation is refused;
• declares an input that accepts what it is called with: { n, i } from a pulse (see below), { at } from a schedule. An operation that declares no input accepts none, and every call would be refused.

No query is sent. An operation that works on entities selects them itself, as expire does.

A cycle is counted from the Unix epoch, not from the start of the process: 300 means :00, :05, :10 of every hour, and 86400 means midnight UTC. For a time of day in a time zone, a weekday or a day of the month, declare a schedule.

§At a time of day

A string is a cron expression:

# components/store.reports/manifest.toa.yaml
name: reports
namespace: store

cadence:
  weekly:
    schedule: 0 9 * * 1 # nine on Monday morning
    zone: Europe/Berlin # in Berlin

operations:
  weekly:
    type: effect
    input:
      type: object
      properties:
        at: { type: integer }
      required: [at]

// components/store.reports/operations/weekly.ts
import type { Context, WeeklyInput } from '../types/index.d.ts'

const WEEK = 7 * 24 * 3600 * 1000

export async function effect ({ at }: WeeklyInput, context: Context) {
  const orders = await context.remote.store.orders.enumerate({
    query: { criteria: `status==approved;CREATED>${at - WEEK};CREATED<${at}`, limit: 1000 }
  })

  const total = orders.reduce((sum, order) => sum + order.total, 0)

  await context.remote.notifications.send({
    input: { to: 'sales@example.com', text: `Sales this week: ${total}` },
    task: true
  })
}

store.reports is a new component. send is the effect of notifications that Call an External Service writes, called here as a task.

at is the moment the call was scheduled for, in milliseconds since the epoch. Compute the period from at, never from the clock: a report made twenty minutes late is still the report for Monday nine.

Without options the expression is the value itself:

cadence:
  report: 0 12 * * 1-5      # noon UTC on weekdays
  invoice: 0 6 1 * *        # six in the morning on the first of the month
  poll: '*/5 * * * *'       # every five minutes
  tick: '*/30 * * * * *'    # every thirty seconds: six fields, the first is seconds

Quote an expression that begins with *; YAML reads a leading * as its own syntax. An expression that does not parse, has another number of fields, or never comes due is refused when the manifest is read.

zone is an IANA time zone, UTC by default. overdue is how many seconds late a call may be and still be made; without it a call is made until the next one comes due.

§What it needs to run

Redis, named in the Context. The replicas agree through it which of them makes a call. The Context toa create wrote names the one npm run dock starts; add the one of a deployment:

# context.toa.yaml
atomicity:
  redis: redis://localhost
  redis@production: redis://redis.example.com

Without it no pulse and no schedule is called, and nothing reports an error. The exception is a pulse declared scope: replica. The annotation is described in Shared Decisions.

The cadence service, for schedules. It is deployed with the application without anything declared. Locally it is added to sys: above. Where the processes are started by hand, toa mono starts it; with toa compose, name it or run it beside:

$ npx toa compose ./components/* --service cadence
$ npx toa serve cadence

See Compositions for where it runs in a deployment.

§Start the service that makes the calls

Declare the schedule, restart, and wait for Monday: weekly is never called, and nothing says why.

$ npm run restart

A pulse is made by the component’s own processes. A schedule is a call that is stored and made later, and what stores and makes such calls is a service of Toa, cadence. The application toa create wrote does not run it. An extension that is declared while its service is not running fails silently: no error at the start and none afterwards.

sys is the process that runs what Toa serves beside the application. Add the service to its line in ecosystem.config.js, and restart:

// ecosystem.config.js
{ ...common, name: 'sys', args: 'serve configuration exposition introspection cadence' },

$ npm run restart

The opposite mistake is reported. A service named on that line while no component declares cadence stops sys when it starts:

Error: TOA_MONGODB_CADENCE_METRONOME is not set.

npm run env writes that variable only once a component declares the extension. Declare first, then name the service.

§Try it

The operation is an ordinary one, so call it with what a pulse or a schedule would send:

$ npm run restart
$ npx toa call store.orders.expire "{ input: { n: 1, i: 0 } }"
$ npx toa call store.reports.weekly "{ input: { at: 1767600000000 } }"

To watch the pulse itself, lower HOUR in expire.ts for a moment, restart, create an order, and observe it by its id after the next five-minute boundary:

$ npm run restart
$ npx toa call store.orders.create "{ input: { customer: 'c1', items: [], total: 1800 } }"
$ npx toa call store.orders.observe "{ query: { id: '7a1c5e0c2b8f4d0e9a3b6c1d2e4f5a6b' } }"

Its status is cancelled. In a test, call the operation directly instead of waiting.

§Write the operation for missed and repeated calls

A pulse can miss an interval, and does not make it up. A deployment, a crash, or an operation that threw costs that interval; nothing is retried. So select by state, not by position: expire cancels every pending order older than an hour, not “the orders that expired in the last five minutes”, and the next call heals a missed one.

A pulse is not called while its previous call is still running. The interval is skipped and the skip is logged; when you see that, the cycle is too short for the work.

A schedule is made late instead of lost. A component that was down on Monday at nine is called when it is back, with the at of Monday nine. Only one occurrence is ever owed: after an outage of a month, one call is made, not four. overdue sets a tighter bound. An occurrence is lost only if the component did not run at all since the occurrence before it.

A schedule can be called twice for one moment. Where a second call would cost something, refuse it by at, for example by recording the at last worked for. A pulse can be called twice only in the rare moment the work changes hands between replicas.

Calls of a schedule can overlap: one may be made while the previous is still running.

A schedule that was changed or removed can be called once more, at the moment already stored for it.

Seconds are coarse. Stored calls are looked over once a minute unless the Context says otherwise, so a schedule that comes due more often than that is made at the next pass; see discreteness.

§Spread the work over a cycle

The full form of a pulse states the cycle and how many intervals it is split into:

# components/store.orders/manifest.toa.yaml
cadence:
  sweep:
    cycle: 86400 # seconds one whole cycle takes
    intervals: 24 # so one call an hour

The gap between calls is cycle / intervals. The operation is called with { n, i }: n is intervals, and i is the interval this call is for, from 0 to n - 1.

// components/store.orders/operations/sweep.ts
import type { Context, SweepInput } from '../types/index.d.ts'

export async function effect ({ n, i }: SweepInput, context: Context) {
  const orders = await context.local.enumerate({
    query: { criteria: `status==pending;shard==${i}`, limit: 1000 }
  })

  // ...
}

An operation that takes i as its share does a twenty-fourth of the work every hour instead of all of it at midnight. How an entity falls into a share is yours to decide; here every order was given a shard from 0 to 23 when it was created.

A missed i = 5 is not covered by i = 6 and waits a whole cycle. Take what in the share is still due, not everything in the share, and a missed interval costs a delay instead of a day’s work.

expire: 300 is short for cycle: 300 with intervals: 1, so expire is called with { n: 1, i: 0 }. intervals cannot exceed cycle.

intervals is not a replica count. With intervals: 24 and three replicas, each replica makes eight of the day’s calls.

§Work of each process

A cache in memory, a pool of connections, a file a process wrote in its own container: each process has to look after its own. Declare scope: replica, and the call is made in every replica:

# components/store.catalog/manifest.toa.yaml
state: ~

cadence:
  trim:
    cycle: 60
    scope: replica

operations:
  trim:
    type: effect
    input:
      type: object
      properties:
        n: { type: integer }
        i: { type: integer }

// components/store.catalog/operations/trim.ts
import type { Context, TrimInput } from '../types/index.d.ts'

export async function effect (input: TrimInput, context: Context) {
  const deadline = context.now() - 600_000

  for (const [key, cached] of context.state.prices)
    if (cached.at < deadline) context.state.prices.delete(key)
}

context.state is described in Keep Something in a Process, where prices is filled. Under scope: replica:

• the operation runs in the replica that made the call;
• it needs no Redis and is called even where atomicity is not configured;
• intervals still splits the cycle over time, in every replica;
• all replicas call at the same moment. An operation that reaches a database or an API reaches it as many times at once as there are replicas, so keep such a pulse to the process’s own work.

§Several entries

An operation with more than one entry takes a list, of either kind:

# components/store.orders/manifest.toa.yaml
cadence:
  sweep:
    - cycle: 3600 # what is due, every hour
    - schedule: 0 3 1 * * # and everything, on the first of the month

Declare an input that accepts both { n, i } and { at }, and tell them apart by what arrived:

// components/store.orders/operations/sweep.ts
import type { Context, SweepInput } from '../types/index.d.ts'

export async function effect (input: SweepInput, context: Context) {
  const everything = input.at !== undefined

  // ...
}

The entries are independent and their calls may overlap. Two schedules of one operation that fall on the same moment are one call.

§Several regions

In an application deployed as several regions, every region makes every pulse and every schedule for itself. An entry that is work for the whole application names the one region that makes it, by rank:

# components/store.reports/manifest.toa.yaml
cadence:
  weekly:
    - schedule: 0 9 * * 1
      zone: Europe/Berlin
      region: 0
    - schedule: 0 9 * * 1
      zone: America/New_York
      region: 1

An entry that names a rank no region has is never called, and nothing says so. An application deployed in one place is rank 0. region cannot be stated beside scope: replica.

§A calendar per item

Every customer is sent a statement on Monday at nine, in the customer’s own time zone. One manifest entry cannot say that: the zone is the customer’s, and so is whether there is a statement at all. The operation that subscribes the customer hands the schedule over with context.schedule:

# components/store.customers/manifest.toa.yaml
cadence: ~

entity:
  properties:
    statement: { type: string } # the id of the schedule

operations:
  subscribe:
    type: transition
    input:
      type: object
      properties:
        zone: { type: string }
  unsubscribe:
    type: transition
  statement:
    type: effect
    input:
      type: object
      properties:
        customer: { type: string }
        at: { type: integer }
      required: [customer, at]

// components/store.customers/operations/subscribe.ts
import type { Context, State, SubscribeInput } from '../types/index.d.ts'

export async function transition (input: SubscribeInput, customer: State, context: Context) {
  customer.statement = await context.schedule(
    'store.customers.statement',
    { input: { customer: customer.id } },
    { schedule: '0 9 * * 1', zone: input.zone }
  )
}

// components/store.customers/operations/unsubscribe.ts
import type { Context, State } from '../types/index.d.ts'

export async function transition (input: null, customer: State, context: Context) {
  await context.schedule.cancel(customer.statement)

  customer.statement = undefined
}

statement is called at every occurrence with the input that was handed over and at beside it, so its input declares both. Compute the week from at, as weekly does above.

The schedule goes on until it is cancelled or until expires, and needs nothing of the component that handed it over to keep running: the cadence service makes the calls.

The third argument is the expression alone, or an object:

overdue is in milliseconds here and in seconds in a manifest: what context is given is in milliseconds, as for context.delay.

Keep the id. context.schedule.cancel(id) is the only way to stop a schedule that has no expires, and nothing lists schedules. To change one, cancel it and hand the new one over.

Handing the same schedule over twice gives one schedule. The same endpoint, request, expression, zone and expires answer the same id, so an operation retried or redelivered does not send two statements every Monday. A schedule cancelled and handed over again comes back under the same id.

What a schedule in the manifest does, this does: a call is made late instead of lost, only one occurrence is ever owed, a call may be made twice for one at, and calls may overlap. Besides:

• a call that was not made does not end the schedule; the next occurrence is made;
• a target that will never take the call ends it: an operation that is gone, or a request its input no longer accepts. It is logged as Delayed call failed with ended: true;
• every call starts a chain of its own;
• occurrences closer together than discreteness are made at most once per pass, the rest are not made;
• it is made by the region that handed it over.

§Reference

cadence:
  <operation>: <seconds>          # a pulse with one interval
  <operation>: <expression>       # a schedule
  <operation>:
    cycle: <seconds>
    intervals: <number>
    scope: group | replica
    region: <rank>
  <operation>:
    schedule: <expression>
    zone: <IANA time zone>
    overdue: <seconds>
    region: <rank>
  <operation>: [<entry>, <entry>]

An entry states either cycle or schedule. One that states both, neither, or a key of the other kind is refused when the manifest is read.

A call of a pulse or a schedule may call anything an ordinary request may. For work that has its own moment per item — this order in an hour — use a delay; for its own calendar, context.schedule. The concepts are in Cadence.