The Same Message Twice

A customer tops up the wallet with 5000 and finds 10000 on it. Nobody wrote “credit twice”; the message that said “credit 5000” simply arrived twice, and the operation did what it was told both times.

This is not a rare accident to be fixed when it shows. Delivery is at least once: an event or a task is delivered until its operation has handled it, and “handled” can only be known for sure by trying again when in doubt. An operation runs again when

• it threw halfway, and the event or task was delivered again;
• its process stopped after the change was stored and before that was acknowledged;
• the reply to a client was lost, and the client pressed the button again;
• a payment provider sent the same notification a second time, a minute or two days later.

The answer is to make the operation idempotent: running it twice with the same message leaves what running it once leaves. There are three ways to get there, and they cover different repeats.

§Set, and look before you write

Most reactions can be phrased as a state to reach rather than a step to take. Then a repeat has nothing to do.

// adds a parcel each time it runs
entry.parcels += 1

// runs twice, ends the same
entry.status = 'shipped'

Where the operation does more than set one value, it first looks whether the work is done. ship is an operation of fulfilment.shipments, which keeps one shipment per order (Messages Out of Order):

// components/fulfilment.shipments/operations/ship.ts
import type { ShipInput, State } from '../types/index.d.ts'

export async function transition (input: ShipInput, entry: State) {
  if (entry.status === 'shipped') {
    entry.DISCARD = true // said again: nothing is written, nothing is announced

    return entry
  }

  if (entry.status !== 'packed') return new Error('NOT_PACKED')

  entry.status = 'shipped'
  entry.carrier = input.carrier

  return entry
}

The repeat is answered as a success — the shipment is shipped, which is what was asked — and DISCARD keeps it from being stored again, so VERSION does not grow and subscribers are not told a second time.

The same idea with a key: keep what was done under the id of whatever caused it. loyalty, the component of Events Worth Subscribing To, keeps the points each order is worth:

// components/loyalty/operations/award.ts
import type { AwardInput, State } from '../types/index.d.ts'

export async function transition (input: AwardInput, entry: State) {
  const before = entry.orders[input.order] ?? 0

  entry.orders[input.order] = input.points
  entry.points = Math.max(0, entry.points + input.points - before)

  return entry
}

“This order is worth 42 points” can be said any number of times. “Add 42 points” cannot.

Taking money is said the same way. debit of billing.wallets (Refuse a Request) takes { amount, ref }, where ref is what the money is for — the order’s id — and the wallet keeps what it took under it, in a property paid:

// components/billing.wallets/operations/debit.ts
import type { DebitInput, State } from '../types/index.d.ts'

export async function transition (input: DebitInput, entry: State) {
  if (entry.paid[input.ref] !== undefined) {
    entry.DISCARD = true // asked again for the same thing: the money is taken

    return { balance: entry.balance }
  }

  if (entry.balance < input.amount) return new Error('INSUFFICIENT_FUNDS')

  entry.balance -= input.amount
  entry.paid[input.ref] = input.amount

  return { balance: entry.balance }
}

“Take the payment for order X” can be said any number of times, by anybody. The wallet remembers the keys for it; what that costs is in A Wallet and Its Ledger.

Prefer this to everything below. It needs no declaration, has no time limit, and survives a repeat that arrives a month later.

§Declare once where the change accumulates

billing.wallets keeps one object per customer, the wallet, with a balance (Describe State). Money given to a customer cannot be phrased as a state to reach: credit adds, the caller does not know the balance, and goodwill money granted twice on purpose is two credits, with nothing to tell them from one credit said twice. Declare once, and the same call arriving again changes nothing and is answered with what the first arrival answered:

# components/billing.wallets/manifest.toa.yaml
operations:
  credit:
    concurrency: retry
    once: true
    query: true
    input:
      type: object
      properties:
        amount: { type: integer, minimum: 1 }
        type: { type: string, enum: [topup, refund, goodwill] }
        ref: { type: string } # what it was for: a checkout, an order
      required: [amount, type, ref]

// components/billing.wallets/operations/credit.ts
import type { CreditInput, State } from '../types/index.d.ts'

export async function transition (input: CreditInput, entry: State) {
  entry.balance += input.amount

  return entry
}

The operation is written as if it ran once. “The same call” is decided for you: an event keeps one identity however many times it is delivered, and a call made from an operation that is delivered again is recognised as the call it made the first time. Two top-ups of 5000 made on purpose are two calls.

A client over HTTP says “this is the same request” itself, with a header. Staff credit a wallet through a route of billing.wallets, open to a role:

# components/billing.wallets/manifest.toa.yaml
exposition:
  /:id/credits:
    auth:role: store:wallets
    io:output: [id, balance]
    PUT: credit

PUT /billing/wallets/8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f/credits/ HTTP/1.1
idempotency-key: 3f1c8a2e-6d0b-4c1f-9a57-0e2b7c9d4a11
content-type: application/yaml

amount: 5000
type: goodwill
ref: welcome

Sent twice, it is answered 200 OK twice with the same balance, credited once.

Everything about the declaration — which operations may have it, what storage it needs, the header — is in Handle Repeated Calls.

§What once does not cover

It remembers a call for an hour, unless the manifest or the Context states another time. That is sized for how late a redelivery arrives, not for how long the money matters: the same top-up reported a day later is a second top-up. The next section is for those.

It prevents the second write, not the second run. A request to a mail provider made from the operation is made on every run.

It knows calls, not meanings. If a bug upstream announces one payment in two different events, those are two calls.

It is not for what a transition that retries calls. A transition declared concurrency: retry is run again when its write is lost, and that attempt makes a new call, not the same one again. The runtime refuses such a call to an operation that declares once (Two Writers at Once). That is why debit, which store.orders.approve calls, is keyed by ref and declares no once, and why credit is reached by an event and by a route.

§Record a fact under the other system’s id

The top-up is paid at a payment provider, and the application learns of it more than once:

• the provider posts a notification, and repeats it — for days — until it is answered with a success;
• the customer’s browser returns from the checkout page and reports the same payment;
• an operator replays a day of notifications after an outage.

There is no single call to remember here. There is a single fact — the provider’s checkout co_1 — and it has an id that every report carries. Keep a record of the fact under that id. Every report reaches the same record; the first one creates it; the rest find it there.

provider's notification ──┐
                          ├──> billing.topups.record     id from the provider's checkout id
customer's return ────────┘      first write creates the record
                                   ~billing.topups.paid               only on creation
                                      └──> billing.wallets.credit   once

# components/billing.topups/manifest.toa.yaml
namespace: billing
name: topups

entity:
  associated: true
  properties:
    checkout: { type: string } # the provider's id of the checkout
    customer: { type: string }
    amount: { type: integer, minimum: 1 }
    currency: { type: string }

operations:
  record:
    concurrency: retry
    query: true
    input:
      type: object
      properties:
        checkout: { type: string }
        customer: { type: string }
        amount: { type: integer }
        currency: { type: string }
      required: [checkout, customer, amount, currency]

associated means the id comes from outside and any id names a record: one that was never written reads as blank, and the first write creates it (Describe State). The id of a record is derived from the provider’s id of the checkout — a hash of it, cut to the length of an id — so the same checkout always names the same record.

// components/billing.topups/operations/record.ts
import type { RecordInput, State } from '../types/index.d.ts'

export async function transition (input: RecordInput, entry: State) {
  if (entry.checkout !== undefined) {
    entry.DISCARD = true // this checkout is already recorded

    return entry
  }

  entry.checkout = input.checkout
  entry.customer = input.customer
  entry.amount = input.amount
  entry.currency = input.currency

  return entry
}

Only the write that created the record announces it:

// components/billing.topups/events/paid.ts
import type { Event } from '@toa.io/core/types'
import type { Entity } from '../types/index.d.ts'

export function condition (event: Event<Entity>) {
  return event.origin === null
}

And the wallet reacts:

# components/billing.wallets/manifest.toa.yaml
receivers:
  billing.topups.paid: credit

// components/billing.wallets/receivers/billing.topups.paid.ts
import type { Entity as Topup } from '@components/billing.topups'

export function request (topup: Topup) {
  return {
    query: { id: topup.customer },
    input: { amount: topup.amount, type: 'topup', ref: topup.checkout }
  }
}

Try it: record the same checkout twice and read the wallet.

$ npm run restart
$ npx toa call billing.topups.record "{ query: { id: '4af147e8801b824e7bd03b000e1d4b64' }, input: { checkout: 'co_1', customer: '8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f', amount: 5000, currency: 'USD' } }"
$ npx toa call billing.topups.record "{ query: { id: '4af147e8801b824e7bd03b000e1d4b64' }, input: { checkout: 'co_1', customer: '8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f', amount: 5000, currency: 'USD' } }"
$ npx toa call billing.wallets.observe "{ query: { id: '8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f' } }"

{ id: '8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f', balance: 5000, VERSION: 1 }

The two mechanisms guard different repeats. The record stops the far duplicate: the notification repeated two days later finds the checkout and announces nothing. once on credit stops the near one: the event billing.topups.paid itself delivered twice.

How the reports reach record is the subject of Receiving Webhooks and Taking a Payment, End to End.

§Why deduplicate where the effect is

It is tempting to stop duplicates at the door: remember the id of every incoming notification and refuse one seen before. That protects against one kind of repeat — the same notification resent — and leaves the others open: the same payment reported by the browser, the internal event delivered twice, the operation run again after a crash. And refusing a notification invites its sender to send it again.

Make the operation that changes the money safe to repeat, and let everything before it repeat freely. The door stores what it was sent; the record keeps one line per fact; the wallet credits once per call. Each is correct alone, so no arrangement of repeats between them can credit twice.

§What can still go wrong

Two reports of one checkout arrive at the same moment. Both read a blank record. concurrency: retry makes the second one run again on what the first stored; it finds the checkout and discards. See Two Writers at Once.

The operation calls an external service. A request to the outside is made again on every run, whatever is declared. Send the provider a key that is the same on every run — the order’s id, not a random value — so that the provider recognises the repeat: Handle Repeated Calls.

A receiver that counts. store.reports counts how often an order changed. There is nothing to set and once is not needed either: pass the order’s VERSION in the request, keep the last one applied, and skip what is not newer; Handle Repeated Calls.

A refusal used as “already done”. An operation that returns an error for a repeat — ALREADY_SHIPPED — is safe behind a receiver, where a returned error ends the work quietly. A client that retried a lost reply, though, is told 422 for a request that succeeded. Where a client may repeat, answer the repeat as a success.

The first run did half. An effect that books a parcel and then stores the booking number can stop between the two. Its repeat must find out what was done — ask the courier by the order’s id — or be written as a multi-step workflow, which goes on from the step that failed.

§Rules of thumb

• Assume every operation behind a receiver, a task or a public route runs twice. Test it: send the same message twice in a scenario.
• Say “make it so”, and look at the state before writing.
• A change that is for something takes that thing’s id and keeps it: one id, one change.
• once for what can only be added and has no such id — together with concurrency: retry, and never called from a transition that retries.
• once is for repeats within its time; for a fact that may be reported again days later, keep a record under the id the other system gave it.
• Announce “created” from origin === null, so that only the first write does.
• Deduplicate at the operation that has the effect, not at the entrance.
• Give an external provider a key that does not change between runs.