VII Patterns

Messages Out of Order

The back office shows a board of orders with their current status, kept by store.board from the events of store.orders, the component Create a Component writes. A customer approves an order and cancels it a second later. The board shows it as approved, and keeps showing it.

§What goes wrong

store.board does the obvious thing, one receiver per event, each setting the status:

# components/store.board/manifest.toa.yaml — do not write it this way
receivers:
  store.orders.approved:
    operation: mark
    arguments: [approved]
  store.orders.cancelled:
    operation: mark
    arguments: [cancelled]

The order was approved and then cancelled. The events were published in that order and handled in the other:

store.orders          store.board
approved  ──┐
cancelled ──┼──────>  cancelled handled: status = cancelled
            └──────>  approved handled:  status = approved      wrong, and it stays wrong

Events and tasks are delivered in no promised order. The first event met a busy process and was taken by another a moment later; or its handling threw once and it was delivered again after a pause, behind the second. Nothing is broken; it is what “delivered until handled” looks like with several processes. Repeats are a special case of the same thing: a repeat is an old message arriving late.

Arrival order is therefore not information. An operation that applies messages as they come is correct only while the system is idle.

There are four ways to be right regardless, and the first one that fits is the one to take.

§Carry the version, and skip what is older

Every stored change of an object increases its VERSION. That number is the publisher’s own clock: of two events about one order, the one with the higher version is the later one, whenever they arrive.

Put it in the payload:

// components/store.orders/events/approved.ts
import type { Event } from '@toa.io/core/types'
import type { Entity } from '../types/index.d.ts'

export function condition (event: Event<Entity>) {
  return event.origin?.status !== 'approved' && event.state.status === 'approved'
}

export function payload (event: Event<Entity>) {
  return {
    id: event.state.id,
    customer: event.state.customer,
    items: event.state.items,
    total: event.state.total,
    status: event.state.status,
    version: event.state.VERSION
  }
}

events/cancelled.ts is the same file with 'cancelled' in its condition, so cancelled carries the same id, customer, total, status and version. The subscriber keeps the last version it applied and skips anything that is not newer:

# components/store.board/manifest.toa.yaml
namespace: store
name: board

entity:
  associated: true # one line per order, under the order's id
  properties:
    status: { type: string }
    total: { type: integer }
    applied: { type: integer }
  blank:
    applied: 0

operations:
  mark:
    concurrency: retry
    query: true
    input:
      type: object
      properties:
        status: { type: string }
        total: { type: integer }
        version: { type: integer }
      required: [status, total, version]

receivers:
  store.orders.approved: mark
  store.orders.cancelled: mark
// components/store.board/receivers/store.orders.approved.ts
// components/store.board/receivers/store.orders.cancelled.ts — the same
export function request (order: { id: string, status: string, total: number, version: number }) {
  return {
    query: { id: order.id },
    input: { status: order.status, total: order.total, version: order.version }
  }
}
// components/store.board/operations/mark.ts
import type { MarkInput, State } from '../types/index.d.ts'

export async function transition (input: MarkInput, entry: State) {
  if (input.version <= entry.applied) {
    entry.DISCARD = true // a repeat, or an event older than what the line already shows

    return entry
  }

  entry.applied = input.version
  entry.status = input.status
  entry.total = input.total

  return entry
}

DISCARD answers without writing, so the line stays as it is.

Try it by delivering the two events in the wrong order yourself:

$ npm run restart
$ npx toa call store.board.mark "{ query: { id: '5b1ad5d8a4c54e1dbb2a4a0b6e1f0c11' }, input: { status: 'cancelled', total: 4200, version: 3 } }"
$ npx toa call store.board.mark "{ query: { id: '5b1ad5d8a4c54e1dbb2a4a0b6e1f0c11' }, input: { status: 'approved', total: 4200, version: 2 } }"
$ npx toa call store.board.observe "{ query: { id: '5b1ad5d8a4c54e1dbb2a4a0b6e1f0c11' } }"
{ id: '5b1ad5d8a4c54e1dbb2a4a0b6e1f0c11', status: 'cancelled', total: 4200, applied: 3, VERSION: 1 }

The same check makes mark safe against repeats, so one line solves both problems.

This works when all the events a subscriber applies to one object come from one object of the publisher: their versions are one sequence. The inherited sync event carries the whole state with its VERSION on every change, and is the usual thing to subscribe to for a copy.

§Treat the event as a signal, and ask

The second way does not apply what the event carries at all. The event only says “this order changed”; the subscriber takes the id from it and reads the current state from the owner.

// components/store.board/receivers/store.orders.sync.ts
import type { Entity as Order } from '@components/store.orders'

export function request (order: Order) {
  return { query: { id: order.id } }
}
// components/store.board/operations/refresh.ts
import type { Context, State } from '../types/index.d.ts'

export async function transition (input: null, entry: State, context: Context) {
  const order = await context.remote.store.orders.observe({ query: { id: entry.id } })

  if (order === null) return new Error('GONE')

  entry.status = order.status
  entry.total = order.total

  return entry
}
# components/store.board/manifest.toa.yaml
operations:
  refresh:
    concurrency: retry
    query: true
    errors: [GONE]

receivers:
  store.orders.sync: refresh

Whatever order the signals arrive in, the last one handled reads the latest state. It is a transition with concurrency: retry on purpose: if two refreshes of one line run together and the one holding the older reading tries to write second, it is run again and reads once more (Two Writers at Once). An effect that observed and then assigned would let the older reading win.

This is the only way when the source has no version to carry — a payment provider’s object, for which notifications arrive in any order and sometimes with the same timestamp (Mirroring a Provider’s Object). Its price is a call per event and a dependency on the owner being reachable when the event is handled.

§Let the value itself say which is newer

Some values can only move one way, and then no version is needed. fulfilment.shipments follows the shipment of an order, one object per order, from what the carrier reports:

# components/fulfilment.shipments/manifest.toa.yaml
namespace: fulfilment
name: shipments

entity:
  associated: true # one shipment per order, under the order's id
  properties:
    status: { type: string, enum: [packed, shipped, delivered] }
    carrier: { type: string }
    booking: { type: string } # the carrier's number of the parcel

operations:
  progress:
    concurrency: retry
    query: true
    input:
      type: object
      properties:
        status: { type: string, enum: [packed, shipped, delivered] }
      required: [status]
// components/fulfilment.shipments/operations/progress.ts
import type { ProgressInput, State } from '../types/index.d.ts'

const STEPS = ['packed', 'shipped', 'delivered']

export async function transition (input: ProgressInput, entry: State) {
  if (STEPS.indexOf(input.status) <= STEPS.indexOf(entry.status)) {
    entry.DISCARD = true // the shipment is already there, or further

    return entry
  }

  entry.status = input.status

  return entry
}

A courier’s “shipped” that arrives after “delivered” changes nothing. The same holds for “the latest expiry date wins” on a membership, or “the highest bid”.

It is the cheapest way and the most limited: it is correct only while the value truly never goes back. A membership whose expiry is brought forward by a refund is refused by “the latest date wins”, and the customer keeps access that was taken away. If the value may ever go back, carry a version.

§Do not create the dependency

Often the order of two messages matters only because both write the same property.

  • Different facts, different properties. shipped sets shipped: <time> and invoiced sets invoiced: <time>. Neither overwrites the other, and no order between them is wrong. A single stage property that both write is what creates the race.
  • A step that needs another to have happened says so. fulfilment.shipments.ship (The Same Message Twice) for a shipment that is not packed returns NOT_PACKED. Behind a receiver that ends the work quietly — which is wrong if packed is merely late. Then throw nothing and refuse nothing: let the later fact be recorded on its own, and derive the status from the facts that are there.
  • Two steps that must run one after the other are calls. When the second really cannot precede the first, make them in one operation, awaited in sequence (Call, Event, or Task).

§Why not rely on arrival order

The order in which messages arrive tells nothing; order them by something the source wrote — its version — or ask the source.

Timestamps taken on arrival do not help: they record the order of arrival. A timestamp taken by the source helps only if the source takes it for every change of that object and no two changes share one; VERSION is exactly that, without a clock. The time at which a subscriber read the source is a valid clock too, and is how a copy of a provider’s object is ordered.

§What can still go wrong

The first event to arrive is not the first that happened. mark meets cancelled for an order it has never seen. That is fine: the line is created by whichever event comes first, and the older one is skipped when it comes. Do not require a “created” event to have arrived.

Events from two objects update one copy. A customer’s line on the board is fed by the customer’s orders and the customer’s wallet. Their versions are unrelated numbers. Keep one applied version per source — applied: { order: 3, wallet: 7 } — or ask.

A skipped message is not an error. Returning an error for an older event is harmless behind a receiver, but do not throw for it: an exception has the event delivered again, to be refused again, until it is set aside for an operator.

The client’s copy. A browser that follows a stream of changes has the same problem and the same cure: expose VERSION, keep the one the copy is at, drop an event that is not newer.

A copy is behind for a moment. Between the change and the event’s handling, the board shows the previous status. A reader that must not act on a stale copy asks the owner instead: Ask the Owner, or Keep a Copy.

§Rules of thumb

  • Never let the last message to arrive win by default.
  • Put VERSION into every payload a subscriber stores something from; keep the last one applied and skip the rest.
  • No version at the source: treat the event as a signal and read the source, in a transition.
  • A value that only moves forward orders itself — as long as it truly never moves back.
  • Different facts go to different properties.
  • Skip an older message without throwing.
Search
Search the documentation