Handle Repeated Calls

A customer must never be charged twice. One rule makes that something you write for:

An operation reached by an event or a task can run again, and occasionally so can any other call. It runs again when it threw, when its process stopped halfway, or when the reply was lost and the caller asked again. The run before may have finished, or may have got halfway.

There are several ways to write for it, from the cheapest. Why calls repeat is explained in Delivery.

§Set instead of add

// runs twice, counts twice
entry.changes += 1

// runs twice, ends the same
entry.status = 'approved'

An operation that only sets values needs nothing else. Most receivers and tasks can be written this way.

§Skip what was already applied

store.reports keeps a sales line per order of store.orders and counts how often the order changed, by its event store.orders.approved. Counting is adding, so the operation has to tell a repeat from a new change. Every event is about one state of one entity, and that state has a VERSION that only grows. Pass it in, and keep the last one applied:

# components/store.reports/manifest.toa.yaml
namespace: store
name: reports

entity:
  properties:
    total: { type: integer }
    changes: { type: integer }
    applied: { type: integer }
  blank:
    total: 0
    changes: 0
    applied: 0

operations:
  add:
    type: transition
    concurrency: retry
    input:
      type: object
      properties:
        total: { type: integer }
        version: { type: integer }
      required: [total, version]

receivers:
  store.orders.approved: add

// components/store.reports/receivers/store.orders.approved.ts
import type { Entity as Order } from '@components/store.orders'

export const request = (payload: Order) => ({
  query: { id: payload.id },
  input: { total: payload.total, version: payload.VERSION }
})

// components/store.reports/operations/add.ts
import type { AddInput, State } from '../types/index.d.ts'

export async function transition (input: AddInput, entry: State) {
  if (entry.applied >= input.version) return entry // a repeat, or an older event

  entry.applied = input.version
  entry.total = input.total
  entry.changes += 1

  return entry
}

Order is the type generated from the entity of store.orders. The same check handles events that arrive out of order, which they may.

§Key the change by what it is for

billing.wallets.debit subtracts an amount from the balance of a customer’s wallet. Subtracting is not setting, and its caller has no version to pass — but it knows what the money is for. debit takes that as ref, the wallet keeps what it took under it, and a second debit for the same thing finds the first:

// components/billing.wallets/operations/debit.ts
import type { DebitInput, State } from '../types/index.d.ts'

export async function transition (input: DebitInput, entry: State) {
  if (entry.paid[input.ref] !== undefined) {
    entry.DISCARD = true // asked again for the same thing: the money is taken

    return { balance: entry.balance }
  }

  if (entry.balance < input.amount) return new Error('INSUFFICIENT_FUNDS')

  entry.balance -= input.amount
  entry.paid[input.ref] = input.amount

  return { balance: entry.balance }
}

paid is a property of the wallet, an object that is {} in the blank, and DISCARD stores nothing. store.orders.approve calls debit with ref: entry.id, “the payment for this order”. However often and for whatever reason the call is made again — a redelivery, a second click, approve run again because its write was lost — the money is taken once.

The operation is safe to repeat by what it is: nothing is declared, no time limit applies, and any caller may call it. What it costs is the keys, which the entity keeps until its own operations remove them. Reach for this wherever the change has a name of its own.

§Declare once

credit of billing.wallets adds an amount to the balance (A Wallet and Its Ledger writes the whole wallet). Money granted by staff has no name to key it by: 500 given twice on purpose is two credits. Declare once, and the same call arriving again changes nothing and is answered with what the first arrival answered. The mechanism is the transactional inbox:

# components/billing.wallets/manifest.toa.yaml
operations:
  credit:
    type: transition
    scope: entry
    concurrency: retry
    once: true
    input:
      type: object
      properties:
        amount: { type: integer, minimum: 1 }
        type: { type: string, enum: [topup, refund, goodwill] }
        ref: { type: string } # what it was for: a checkout, an order
      required: [amount, type, ref]

The code of the operation says nothing about it:

// components/billing.wallets/operations/credit.ts
import type { CreditInput, State } from '../types/index.d.ts'

export async function transition (input: CreditInput, entry: State) {
  entry.balance += input.amount

  return entry
}

When an event that a receiver binds to credit is delivered twice, the balance moves once and both deliveries get the same reply.

§Try it

A request may state the id of the call itself, as id beside input and query. Send the same one twice:

$ npm run restart
$ npx toa call billing.wallets.credit 
    "{ id: aa11e57cc0e14fce95c4496c21086781, query: { id: '8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f' }, input: { amount: 500, type: goodwill, ref: welcome } }"
$ npx toa call billing.wallets.credit 
    "{ id: aa11e57cc0e14fce95c4496c21086781, query: { id: '8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f' }, input: { amount: 500, type: goodwill, ref: welcome } }"

Both print the same wallet, credited once. With another id, or with none, the second call credits again. Stating id is for trying things out; inside an operation leave it to the runtime.

§Which calls are the same call

You do not write call ids:

• A call that starts from the gateway, a receiver or a task gets an id once, and keeps it however many times it is delivered.
• A call made from an operation takes its id from the call that operation is serving. An effect that calls credit, delivered twice, calls it under the same id both times.

The id is not the content: two credits of the same amount made on purpose are two calls.

So that a second run lines up with the first, make the same calls in the same order for the same input. An operation that branches on something that changed between two runs makes different calls, and the repeat is not recognized.

A caller that does not declare once itself is not a problem for its callees: delivered again, it makes the same call, and credit recognizes it.

§Not from a transition that retries

Declared concurrency: retry, a transition is run again when its write is lost, and that second attempt is not the same delivery: its call has an id of its own, and once would apply it once per attempt. So the runtime refuses the call where it is made, on the first attempt as on any other:

UnrepeatableException: 'billing.wallets.credit' is made once, and a transition retried on a lost write would make it again

What such a transition calls is made safe to repeat by what it is, and then it needs no once: that is why debit, which approve calls, declares none. Where the call cannot be keyed, it is made by a receiver of the event the transition’s change publishes, or the two are called from an effect. concurrency: none on the transition lifts the refusal too, and is rarely right: every conflict becomes a failure its caller has to handle.

§Which operations can declare it

It needs MongoDB running as a replica set or a sharded cluster. A component that declares once on any other storage does not start, and says which component and which storage.

An operation that declares once refuses a call that carries no id.

§How long a call is remembered

once: true remembers a call for an hour, unless the Context states another time for the whole application:

# context.toa.yaml
inbox:
  retention: 3600 # seconds

An operation whose callers may retry later states its own, in seconds:

# components/billing.wallets/manifest.toa.yaml
operations:
  credit:
    once: 86400 # a client may retry a credit for a day

The operation’s number wins. Choose it by how late a repeat can arrive, not by how long the change matters. The minimum is 600 in both places; a manifest or a Context that states less is refused. A repeat that arrives after the time has passed is a new call.

§What once does not cover

A call that changed nothing is not remembered. If the operation returned an error, threw, or discarded its change, the same call made again runs again, and answers the same way.

The code can still run more than once. once prevents the second write to the entity, not the second run. So whatever the operation does besides changing its entity happens on every run:

export async function transition (input: CreditInput, entry: State, context: Context) {
  entry.balance += input.amount

  // made on every run, also on the run of a repeat
  await context.remote.notifications.send({
    input: { to: entry.id, text: `${input.amount} was added to your wallet` }
  })

  return entry
}

Here the mail is sent on every run, and once on what is called could not help: credit retries. Leave such a consequence to a receiver of an event of the wallet. A request to an external service is made again in the same way; see below.

It holds within one region. In an application deployed in several regions, a repeat that arrives in another region is applied there.

§Retries from HTTP clients

A client that retries a request it never saw answered says it is the same request with a header:

POST /store/orders/7a1c5e0c2b8f4d0e9a3b6c1d2e4f5a6b/approval/ HTTP/1.1
idempotency-key: 3f1c8a2e-6d0b-4c1f-9a57-0e2b7c9d4a11

The client generates the key and repeats it only to retry the same request. A retry with the same key reaches the operation as the same call and, where the operation declares once, is answered with the first reply. The key counts per caller, method and path; the body is not part of it, so the same key with another body gets the first reply.

Three things to know:

• On a route nothing authenticates, two clients that pick the same key collide. Put authentication on a route that changes state.
• A key sent to a method whose operation does not declare once is accepted and does nothing.
• A method says whether it is safe to retry this way in what it answers to OPTIONS; see Discovery.

The header is described with the rest of the protocol.

§Keys for external services

An email, a payment through a provider, any request to the outside is made again in full on every run. Either make it safe to repeat, or give the provider a key to recognize the repeat by; most payment and messaging services accept one:

// components/payments.gateway/operations/charge.ts
import type { ChargeInput, Context } from '../types/index.d.ts'

export async function effect (input: ChargeInput, context: Context) {
  const response = await context.fetch(`${context.configuration.url}/charges`, {
    method: 'POST',
    headers: { 'idempotency-key': input.order },
    body: JSON.stringify({ amount: input.amount })
  })

  // ...
}

Use something that is the same on every run: the id of the order, not a new random value. An effect that makes several such requests in a row can be declared resumable, and then context.id() gives a key that stays the same across its runs; see Write a Multi-Step Workflow.

§When it keeps failing

An event or a task whose operation keeps throwing is retried a few times over a couple of minutes and then set aside for an operator, who puts it back once the cause is fixed; see Operating. An operation written by the rules above takes it back safely.

§Reference