VII Patterns

Receiving Webhooks

A customer tops up their shop balance on the payment provider’s page. The provider tells the shop about it by a webhook: an HTTP request it sends to an address of the shop, some time after the payment. The shop must add the money to the customer’s wallet — once.

The naive way is a route that reads the body and credits the wallet. Four things break it:

  • Anyone can send that request. The address is public; a forged “paid 5000” is free money.
  • The provider sends the same notification again. It repeats until it is answered with a success, and providers such as Stripe keep trying for days. Every repeat credits again.
  • The same payment is reported more than once on purpose: “checkout completed”, then “payment succeeded”, and the customer’s browser reports it too.
  • The process may stop between storing something and crediting. The provider got its 2xx and will never ask again; the money is lost.

§Verify, store, announce — and deduplicate where the money moves

provider ── POST /payments/webhooks/ ──> payments.webhooks.ingest ── call ─> payments.gateway.decode
                                           │ stored: one object per delivery
                                           ~ payments.webhooks.checkout
                                           v
                                       billing.topups.record      one object per checkout, ever
                                           ~ billing.topups.paid  only when it was created
                                           v
                                       billing.wallets.credit     once

A ──> is a call, a ~ is an event, which a receiver binds to an operation. Three components, each with one job: keep what the provider said, keep what was paid, keep the balance. The third, billing.wallets, is the wallet of A Wallet and Its Ledger; payments.gateway, with its decode, is the gateway to the provider.

§The door

# components/payments.webhooks/manifest.toa.yaml
name: webhooks
namespace: payments

entity:
  properties:
    event: { type: string }      # the provider's id of the notification
    type: { type: string }
    created: { type: integer }
    checkout: { type: object }
  required: [event, type, created]

operations:
  ingest:
    query: false
    errors: [DECODE]
    input:
      type: object
      properties:
        payload: { type: string }
        signature: { type: string }
      required: [payload]

exposition:
  /:
    POST:
      anonymous: true
      map:buffer: payload
      map:headers:
        signature: x-signature
      io:output: false
      endpoint: ingest
// components/payments.webhooks/operations/ingest.ts
import type { Context, IngestInput, State } from '../types/index.d.ts'

export async function transition (input: IngestInput, entry: State, context: Context) {
  const notification = await context.remote.payments.gateway.decode({
    input: { payload: input.payload, signature: input.signature ?? '' }
  })

  if (notification instanceof Error) {
    context.logs.error('Webhook refused', { code: notification.code })

    return notification
  }

  entry.event = notification.id
  entry.type = notification.type
  entry.created = notification.created

  if (notification.type === 'checkout.completed') entry.checkout = notification.data
}
  • The route is open to everyone: the provider has no account in the shop. What stands in for authentication is the signature.
  • map:buffer hands over the body exactly as it was sent, because the signature is computed over those bytes. map:headers adds the signature header.
  • Verifying is a call to the gateway, which holds the signing secret: ingest cannot decide whether to store anything without the answer. A forged body is the returned error DECODE; nothing is stored, and the sender gets 422.
  • io:output: false: the provider needs a status, not a body. A stored notification answers 201.
  • The refusal is logged with its code and without the body: an unverified body is not something to keep.

A notification of a type the shop does nothing with is stored all the same and answered 201. Refusing it would only make the provider send it again.

§The announcement

// components/payments.webhooks/events/checkout.ts
import type { Event } from '@toa.io/core/types'
import type { Entity } from '../types/index.d.ts'

export function condition (event: Event<Entity>) {
  return event.origin === null && event.state.checkout !== undefined
}

payments.webhooks.checkout is published for a stored notification that carries a checkout, with the stored object as its payload. The event is recorded together with the notification, in one write. If the process stops right after answering the provider, the event is still published when a process is back, and whoever handles it is retried until it succeeds (Transactional Outbox).

That is why the answer to the provider can be given before any money has moved: once the notification is stored, the rest will happen.

§One record per payment

# components/billing.topups/manifest.toa.yaml
name: topups
namespace: billing

entity:
  associated: true
  properties:
    checkout: { type: string }   # the provider's id of the checkout
    customer: { type: string }
    amount: { type: integer, minimum: 1 }
    currency: { type: string }
  required: [checkout, customer, amount, currency]

operations:
  record:
    concurrency: retry
    query: true
    input:
      type: object
      properties:
        checkout: { type: string }
        customer: { type: string }
        amount: { type: integer }
        currency: { type: string }
      required: [checkout, customer, amount, currency]

receivers:
  payments.webhooks.checkout: record
// components/billing.topups/lib/key.ts
import { createHash } from 'node:crypto'

// the id of the record of a checkout: the same checkout always gives the same id
export function key (checkout: string) {
  return createHash('sha256').update(checkout).digest('hex').slice(0, 32)
}
// components/billing.topups/lib/purchase.ts
import { key } from './key.ts'

export interface Checkout {
  id: string
  status: string
  amount: number
  currency: string
  metadata: { customer: string }
}

export function paid (checkout?: { status?: string, metadata?: { customer?: string } }) {
  return checkout?.status === 'paid' && checkout.metadata?.customer !== undefined
}

export function request (checkout: Checkout) {
  return {
    query: { id: key(checkout.id) },
    input: {
      checkout: checkout.id,
      customer: checkout.metadata.customer,
      amount: checkout.amount,
      currency: checkout.currency
    }
  }
}
// components/billing.topups/receivers/payments.webhooks.checkout.ts
import type { Checkout } from '../lib/purchase.ts'
import { paid, request as purchase } from '../lib/purchase.ts'

export const condition = (notification: { checkout: Checkout }) => paid(notification.checkout)

export const request = (notification: { checkout: Checkout }) => purchase(notification.checkout)
// components/billing.topups/operations/record.ts
import type { RecordInput, State } from '../types/index.d.ts'

export async function transition (input: RecordInput, entry: State) {
  if (entry.checkout !== undefined) {
    entry.DISCARD = true // recorded before: nothing to write

    return entry
  }

  entry.checkout = input.checkout
  entry.customer = input.customer
  entry.amount = input.amount
  entry.currency = input.currency

  return entry
}
// components/billing.topups/events/paid.ts
import type { Event } from '@toa.io/core/types'
import type { Entity } from '../types/index.d.ts'

export function condition (event: Event<Entity>) {
  return event.origin === null
}

The entity is associated: its id is not generated, it is derived from the provider’s id of the checkout. Every report of one checkout — the first delivery, a repeat three days later, another notification type, the customer’s browser — is addressed to the same object. The first one creates it. Every later one finds checkout already set and writes nothing, so no event follows.

billing.topups.paid is published when the record is created (origin === null), which happens once per checkout, ever. That event is what credits the wallet:

# components/billing.wallets/manifest.toa.yaml
receivers:
  billing.topups.paid: credit
// components/billing.wallets/receivers/billing.topups.paid.ts
import type { Entity as Topup } from '@components/billing.topups'

export function request (topup: Topup) {
  return {
    query: { id: topup.customer },
    input: { amount: topup.amount, type: 'topup', ref: topup.checkout }
  }
}

credit declares once; the wallet’s side is in A Wallet and Its Ledger. Topup is the type generated from the entity of billing.topups.

§Try it

With an empty provider key the gateway’s stub accepts an unsigned body. Restart, and send the same notification twice:

$ npm run restart
$ curl -i http://localhost:8000/payments/webhooks/ -d '{
    "id": "evt_1", "type": "checkout.completed", "created": 1760000000,
    "data": { "id": "co_1", "status": "paid", "amount": 5000, "currency": "USD",
              "metadata": { "customer": "8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f" } } }'

Both are answered 201 Created, and the wallet was credited once:

$ npx toa call billing.wallets.observe "{ query: { id: '8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f' } }"
{ id: '8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f', balance: 5000, held: 0, VERSION: 1 }

§Why the record, when credit is declared once

They stop different repeats.

once recognizes the same call arriving again: the event billing.topups.paid delivered a second time to the wallet. It remembers a call for an hour by default. It cannot recognize the provider’s repeat, because that is a new HTTP request, stored as a new notification and announced by a new event — a different call that happens to say the same thing, possibly days later.

The record recognizes the same payment, however and whenever it is reported, for as long as the record exists. Both techniques are set side by side in The Same Message Twice.

Deduplicate a repeat from the outside world by an object stored under the outside world’s id, and announce the consequence only when that object is created. Use once for the repeat of your own call.

And why not refuse a repeated notification at the door, by its event id? Because it would not be enough — the same payment arrives under several notification ids, and from the browser with none — and because a refusal is a non-2xx answer, which the provider answers by sending the notification again. The door stores everything; the place where the money moves decides what is new.

§Refusing on purpose

The provider repeats a notification only when it is answered with something other than a success. That makes the answer a tool: if handling a notification needs something that is not available right now, refuse it, and it comes back.

The top-up above needs nothing but the shop’s own storage, so it never refuses a genuine notification. A notification that says “this membership changed” is different: the shop reads the membership from the provider before it answers, and if the provider cannot be read it returns the gateway’s UNAVAILABLE. That is in Mirroring a Provider’s Object.

Do that work by a call inside ingest, before the reply — not in a receiver. A receiver runs after the provider has been told 201, and a returned error there is final.

§What can still go wrong

A forged request. decode refuses it; nothing is stored and nothing follows.

A repeat, minutes or days later. Stored as another notification, announced, addressed to the same record, which is left as it is. No second paid, no second credit.

Two reports of one checkout at the same moment. Both read an empty record and both try to create it. One write wins; concurrency: retry runs the other again, and this time it finds the record and writes nothing (Two Writers at Once).

Notifications out of order. A top-up is a single fact and has no order to get wrong. An object that keeps changing at the provider does, and needs more.

billing.wallets is down when the top-up is recorded. billing.topups.paid waits and is delivered when it is back. The provider was answered long ago and is not involved.

billing.topups is down when the notification is stored. The same, one step earlier.

The webhook never arrives. The provider’s address for the shop was misconfigured for a day, or the shop was down longer than the provider keeps trying. Nothing in the shop can notice a message it never got; it has to ask. That is the second route of Taking a Payment, End to End.

A notification without a customer. paid() is false for a checkout that carries no metadata.customer — one created by hand in the provider’s dashboard, say. It is stored and nothing follows, which is what should happen to a payment the shop cannot attribute.

§Rules of thumb

  • Take the body with map:buffer; verify the signature in the gateway, by a call, before storing anything.
  • Store every verified notification, and answer with a success even when you do nothing with it.
  • Do after the reply, by events, everything that needs only your own application.
  • Do before the reply, by a call, whatever might need the provider to send the notification again — and return an error when it does.
  • Keep one object per thing that happened at the provider, under an id derived from the provider’s id; write it once.
  • Announce “created”, and hang the consequences on that event.
  • Never make a notification’s handler add to something without such a record in front of it.
  • Raw notifications pile up; an expiry index removes old ones.
Search
Search the documentation