VII Patterns

Two Writers at Once

One kettle is left in stock. Two customers press “Buy” within the same second. The shop must sell it to one of them and tell the other it is gone.

§What goes wrong

The natural first version reads, checks, and writes:

// components/fulfilment.stock/operations/reserve.ts — do not write it this way
import type { Context, ReserveInput } from '../types/index.d.ts'

export async function effect (input: ReserveInput, context: Context) {
  const item = await context.local.observe({ query: { id: input.product } })

  if (item.available < input.quantity) return new Error('OUT_OF_STOCK')

  await context.local.assign({
    query: { id: input.product },
    input: { available: item.available - input.quantity }
  })
}

Run it twice at once:

request A                         request B
reads available: 1
                                  reads available: 1
checks 1 >= 1, passes
                                  checks 1 >= 1, passes
writes available: 0
                                  writes available: 0

Both customers are told the kettle is theirs. Each request was correct against what it read; what it read was no longer true when it wrote. The same shape gives a balance spent twice, two couriers assigned to one parcel, and a counter that loses an increment.

An application runs in several processes, and each process serves many requests at a time, so “at once” is the normal case. A mutex inside one process changes nothing: the other request is in another process.

§The solution

Put the check and the write into one transition over the object that holds the fact, and declare concurrency: retry:

# components/fulfilment.stock/manifest.toa.yaml
namespace: fulfilment
name: stock

entity:
  associated: true # one object per product, under the product's id
  properties:
    available: { type: integer, minimum: 0 }
  blank:
    available: 0

operations:
  reserve:
    concurrency: retry
    query: true
    input:
      type: object
      properties:
        quantity: { type: integer, minimum: 1 }
      required: [quantity]
    errors: [OUT_OF_STOCK]
  transit: # how the warehouse says what it has
    concurrency: retry
    input:
      type: object
      properties:
        available: { type: integer, minimum: 0 }
      additionalProperties: false
// components/fulfilment.stock/operations/reserve.ts
import type { ReserveInput, State } from '../types/index.d.ts'

export async function transition (input: ReserveInput, entry: State) {
  if (entry.available < input.quantity) return new Error('OUT_OF_STOCK')

  entry.available -= input.quantity

  return entry
}

A transition is given the object as it is, and what it leaves is stored — but only if nobody changed the object in between. If somebody did, concurrency: retry has the function run again on the fresh object:

request A                         request B
given available: 1
                                  given available: 1
leaves 0, stored
                                  leaves 0, NOT stored: the object has changed
                                  run again, given available: 0
                                  returns OUT_OF_STOCK

Nothing is locked, and nobody waits for anybody. The loser simply decides again from the truth.

Try it: put one unit in stock with the inherited transit, and send two reservations of it at the same time:

$ npm run restart
$ npx toa call fulfilment.stock.transit "{ query: { id: 'c0ffee00c0ffee00c0ffee00c0ffee00' }, input: { available: 1 } }"
$ npx toa call fulfilment.stock.reserve "{ query: { id: 'c0ffee00c0ffee00c0ffee00c0ffee00' }, input: { quantity: 1 } }" &
$ npx toa call fulfilment.stock.reserve "{ query: { id: 'c0ffee00c0ffee00c0ffee00c0ffee00' }, input: { quantity: 1 } }" &

One prints the item with available: 0, the other an error with the code OUT_OF_STOCK.

store.orders.create, the operation that makes an order (Add Configuration writes it as a transition), then makes the reservation by a call, because its answer decides whether there is an order at all (Call, Event, or Task).

The declaration is described in Write Operations.

§Why a transition

A decision that depends on the current state is made in a transition over the object that holds that state, with concurrency: retry.

  • An effect that reads and then writes makes two separate steps, and anything fits between them.
  • An assignment never reads, so it never conflicts — and it cannot check anything. It is right for “set the note to this”, and wrong for “take one if any are left”.
  • A transition reads and writes as one step that either happens against the state it saw or is made again.

concurrency: none instead of retry fails the losing call with an exception — 409 over HTTP — and leaves the retry to the caller. It is rarely what a business rule wants; retry is the default to reach for.

§Keep the fact in one object

The technique protects one object. A rule over several — “the sum of all reservations of this product does not exceed what the warehouse has” — cannot be checked this way: two reservations are two objects, both are stored, and neither conflicts with the other.

So keep the number the rule is about as a property of one object, and change it only by transitions: available on the stock item, balance on the wallet. Do not compute a balance by adding up entries each time it is needed; store it, and let the entries be the record of how it came about (A Wallet and Its Ledger).

§One object per thing that changes on its own

The opposite mistake is one object for too much. A single warehouse object with a dictionary of every product makes each sale conflict with every other sale, and under load the transition is run again and again. One stock item per product conflicts only when two customers want the same product, which is exactly when a conflict means something.

§What a second run repeats

A transition under concurrency: retry can run more than once for one call. Whatever it does to its own object is simply done again on fresh state. Everything else it does is done again as well:

// components/store.orders/operations/approve.ts
import type { Context, State } from '../types/index.d.ts'

export async function transition (input: null, entry: State, context: Context) {
  if (entry.status === 'approved') {
    entry.DISCARD = true // asked again: it is approved already

    return entry
  }

  if (entry.status !== 'pending') return new Error('NOT_PENDING')

  // made on every attempt
  const debit = await context.remote.billing.wallets.debit({
    query: { id: entry.customer },
    input: { amount: entry.total, ref: entry.id }
  })

  if (debit instanceof Error) return debit

  entry.status = 'approved'

  return entry
}

This is approve of store.orders as Call, Event, or Task leaves it, declared concurrency: retry: billing.wallets.debit takes the total off the balance of the customer’s wallet, and DISCARD answers a repeated request without writing.

Two clicks on “Approve” arrive together. Both runs see pending, both call debit, one stores approved; the other loses its write, is run again, sees approved and writes nothing — after its first attempt has called debit too. The wallet was asked twice.

§Key the call by what it is for

It took the money once, because the call says what it is for. ref is the order’s id — “the payment for order X” — and the wallet keeps what it took under it:

// components/billing.wallets/operations/debit.ts
import type { DebitInput, State } from '../types/index.d.ts'

export async function transition (input: DebitInput, entry: State) {
  if (entry.paid[input.ref] !== undefined) {
    entry.DISCARD = true // asked again for the same thing: the money is taken

    return { balance: entry.balance }
  }

  if (entry.balance < input.amount) return new Error('INSUFFICIENT_FUNDS')

  entry.balance -= input.amount
  entry.paid[input.ref] = input.amount

  return { balance: entry.balance }
}

paid is a property of the wallet, an object that starts as {}; A Wallet and Its Ledger has the manifest and what the keys cost. A debit keyed this way cannot be taken twice, whoever asks and why: a second attempt, a second click, a message delivered again.

What a transition that retries calls is safe to call again by what it is. Reach for this first. Where the thing asked for can also be taken back — money set aside for an order that is then cancelled — the same key undoes it: Reservations.

§once does not help

once makes the runtime apply a call once however many times it is delivered. An attempt made because a write was lost is not a redelivery: it makes a new call, with an id of its own, and once would apply it once per attempt. So the runtime refuses a call from a transition that retries to an operation that declares once, the first time it is made. Had debit declared it:

UnrepeatableException: 'billing.wallets.debit' is made once, and a transition retried on a lost write would make it again

once stays right for an operation that accumulates, has nothing to be keyed by, and is not called from a transition that retries: the wallet’s credit, reached by an event and by a staff route (The Same Message Twice).

Where a call cannot be keyed, there are three other ways to write the transition:

  • Leave the call to an event. The transition only changes its own object, and a receiver of the event that change publishes makes the call: once, for the attempt that was stored.
  • Call both from an effect, which has no write of its own to lose.
  • concurrency: none. The attempt that loses its write fails with 409 and is not run again. It is rarely right: the conflict becomes the caller’s to handle, and two clicks still make two calls.

Reading from another component inside a transition is always safe: a second run reads again.

Two smaller consequences of a second run:

  • The clock and random values differ between runs. Decide from the state and the input.
  • Do not send, count or log business facts from inside a transition. Leave a consequence to an event, which is published once, for the run that was stored.

§The other kind of lost update

concurrency covers changes that happen while an operation runs. A different race takes minutes: two members of staff open the same order, both edit the delivery address, and the second “Save” silently undoes the first. Each save is a correct operation on current state; what is stale is the form on the screen.

Here the caller says which version it saw, and the write is refused if the object has moved on:

await context.remote.store.orders.transit({
  query: { id: order.id, version: order.VERSION },
  input: { address }
})

Over HTTP it is the if-match header, and the refusal is 412; the client reads again and shows the user what changed. See Call Operations and Select State and Cache Replies and Prevent Lost Updates.

Use the operation’s concurrency for rules, and the caller’s version for edits by people.

§When a lock is the answer

A version protects an object of the application. It cannot protect what is outside: a parcel booked at the courier, a batch submitted to the bank. Work that reaches outside and must not overlap with itself takes a lock, which only one process holds at a time. fulfilment.shipments keeps one shipment per order, under the order’s id (Messages Out of Order), and books the carrier for it:

// components/fulfilment.shipments/operations/book.ts
import type { BookInput, Context } from '../types/index.d.ts'

export async function effect (input: BookInput, context: Context) {
  return await context.atom.lock(`booking:${input.order}`, async () => {
    const shipment = await context.local.observe({ query: { id: input.order } })

    if (shipment.booking !== undefined) return shipment // whoever held the lock before did it

    const booking = await book(shipment, context)

    return await context.local.assign({ query: { id: input.order }, input: { booking } })
  })
}

Two rules make a lock useful:

  • Read the state again inside it. Whoever waited for the lock must see what the previous holder did; a decision made before the lock is the stale read all over again.
  • Do not lock around changes of an object. A transition already handles them, without making anybody wait.

See Coordinate Replicas.

§What can still go wrong

The loser’s answer surprises the client. The second customer gets OUT_OF_STOCK although the page showed one kettle. That is the correct outcome of a race; make sure the code is declared in errors, and that the client says something better than “error”.

The check is in the wrong component. store.orders observes the stock, sees one kettle, then calls reserve. The observation was only advice: the decision is reserve‘s. Drop the preliminary check, or keep it knowing that the refusal can still come.

A hot object. Every sale of the season’s bestseller changes one stock item, and runs pile up. A transition is run again a limited number of times and then fails with an exception. If that happens, the object holds too much — split it, per warehouse or per batch.

A transition that takes long. The longer it runs, the likelier somebody else writes first. A transition that waits ten seconds for an external service loses most races and repeats the request each time. Do the slow part in an effect, and keep the transition to the decision: Reply Before the Work Is Done.

§Rules of thumb

  • “Read, check, write” is one transition with concurrency: retry — never an effect, never two calls.
  • The check lives in the component that owns the number.
  • One object per thing that changes on its own; a number a rule is about is stored, not summed.
  • Assume a transition runs twice: what it calls is keyed by what it is for, and consequences go to events.
  • A transition that retries does not call an operation that declares once.
  • People editing a form: the caller’s version or if-match.
  • A lock only for work outside the application, with the state read again under it.
Search
Search the documentation