VI Shipping

Run in More Than One Region

The shop gets customers on another continent, and they should be served from a deployment near them — with the same orders and accounts visible in both places. Deploy the application once per place, each with its own database, and declare each deployment a region: regions carry their changes to each other until all hold the same data. Regions explains the model and what it guarantees.

§Declaring regions

A region is an environment that declares convergence in the Context:

# context.toa.yaml
mongodb@eu: mongodb://mongo.eu.example.com
mongodb@us: mongodb://mongo.us.example.com

amqp@eu: amqp://rmq.eu.example.com
amqp@us: amqp://rmq.us.example.com

convergence@eu:
  priority: 0
  binding:
    provider: amqp
    pointer: amqp://cnv.eu.example.com

convergence@us:
  priority: 1
  binding:
    provider: amqp
    pointer: amqp://cnv.us.example.com
$ npx toa deploy eu
$ npx toa deploy us

Each region has its own database, its own broker, and one more broker used only for carrying changes between regions.

  • priority — the region’s rank. 0 outranks 1. It decides which of two concurrent writes wins.
  • binding.provider — amqp.
  • binding.pointer — the address of this region’s convergence broker: one URL or a list of them. It carries no credentials; they are a secret, listed by toa export secrets.

An environment without convergence is not a region. The same Context still deploys to staging as a single place.

Nothing is declared in components. Every component that stores state converges, including the ones extensions bring: an account registered in one region can sign in to another.

§Ranks

Every region needs a distinct rank, and nothing checks it at deploy. Two regions with the same rank cannot settle a tie, and the only sign is an error in the log when a record arrives carrying the receiving region’s own rank. Keep all convergence@... blocks together in the Context so that the table is visible.

An application that already runs becomes rank 0. The environment you turn into the first region takes priority: 0.

§What it requires

  • The database must be a replica set. In a region, a component with state does not start without one.
  • atomicity.redis must be set in the Context; see Shared Decisions. Without it a change that failed to publish is never retried, and the log says so repeatedly.
  • Every region runs the same components. A component deployed in one region and not another has nowhere to send its changes there; they are logged and dropped.
  • A storage that cannot converge is reported at start. The component runs, but its state stays regional, and one line in the log says so. Look for it after the first deployment.

Only what happens after convergence is on is carried. Existing data is copied when a region is added.

§Two regions change the same entity

An order is at version 3 in both regions. In eu a customer cancels it; in us, at the same moment, an operator approves it. When the two changes cross, both regions keep the one written in eu, rank 0.

The losing write is discarded whole. Nothing is merged: the approval is gone, with everything else that write changed. Its events are not undone: store.orders.approved was published in us and its receivers ran.

A write that is later than another — a higher version of the entity — wins regardless of rank. Rank settles only writes made at the same version.

§Writing components for several regions

Most components need no change. These are the cases that do.

§Keep one writer per entity where you can

A write can be lost only when the same entity is written in two regions at nearly the same time. If each customer is served by one region — their home region — an order is never written in two places at once. That is a routing decision made outside the application, in whatever directs clients to a region.

Where two regions can legitimately write the same entity, design the entity so that losing one write whole is acceptable, or split it so that each part has one writer.

§Do not rely on unique indexes other than id

A unique index is enforced within one database. Two regions can each accept a product with the same sku before hearing from the other. The record that arrives second cannot be stored in the other region; it is logged and dropped, and the regions differ for that entity.

Identifiers generated by the runtime do not collide. If a value must be unique across regions, make it the entity’s id, or allocate it in one region only.

§Reads may be behind

A read in one region may not yet see a write made in another, and while the link between regions is down the delay is unbounded. A client that writes in one region and immediately reads in another may not find what it wrote. Nothing is transactional across regions.

§once is regional

An operation declared with once (Handle Repeated Calls) recognises a repeated call in the region where it was first made. Retried against another region, it runs again. Keep a client’s retries in one region.

§Scheduled work

A delayed call is made only by the region that asked for it. Pulses and schedules are made by every region, each for itself, unless the entry names the region that makes it. Run Work on a Schedule shows region on an entry, and Do Something Later shows cadence.regions, which also lets a surviving region take over the scheduled work of a lost one.

§Telling where you are

context.region is the rank of the region the operation runs in. It is 0 where the application is deployed as a single place. The approve of store.orders that Create a Component writes could keep it in a property added to the entity, approvedIn:

// components/store.orders/operations/approve.ts
import type { Context, State } from '../types/index.d.ts'

export async function transition (input: null, entry: State, context: Context) {
  entry.approvedIn = context.region
  // …
}

Each entity also records the region that last wrote it, in the system property REGION.

§Adding a region

The shop runs in eu and is to run in us as well. Follow the steps in this order: it is what keeps changes from being lost between copying the data and starting the new region.

§Before you start

  • eu must be on a replica set with atomicity configured. Turning convergence on for an application that is not on a replica set takes it down, since its components refuse to start.
  • Choose ranks: eu is 0, us is 1.

§1. Stand up the new region’s infrastructure

Its database, its application broker, its convergence broker. Deploy nothing onto it yet.

Declare both regions in the Context, and deploy the existing deployment as eu, with its own convergence broker, so that it publishes its changes.

§2. Federate the convergence brokers, both ways

Toa does not configure the link between brokers. On each region’s convergence broker, add an upstream for every other region and one policy. On the us broker:

$ rabbitmqctl set_parameter federation-upstream eu 
    '{"uri":"amqp://cnv.eu.example.com","exchange":"convergence.out","max-hops":1}'

$ rabbitmqctl set_policy convergence '^convergence.in$' 
    '{"federation-upstream-set":"all"}' --apply-to exchanges

and the mirror of it on the eu broker, with an upstream named us.

It is one link per other region, whatever the number of components. The eu broker will log that what it federates from does not exist yet; that is expected until the next step.

§3. Declare the new region’s queues

Do this before copying any data. Until the new region has queues, the old region’s changes are dropped, not kept. From the moment the queues exist, every change is held in them.

toa export convergence prints exactly what the broker must hold:

$ npx toa export convergence us 
    | curl -u <user>:<password> -H 'content-type: application/json' 
           -X POST --data @- http://<us-convergence-broker>:15672/api/definitions

The import only adds, so it is safe to run twice. Where the broker’s management API is out of reach, --format commands prints rabbitmqadmin commands to run instead.

Do not change the arguments of what it declares. The application declares the same queues when it starts and fails if they differ.

Check that the link picked them up. On the eu broker:

$ rabbitmqctl list_queues name messages | grep federation

§4. Copy the data

Dump the eu database and restore it into the us database, after step 3. Changes made while the dump was being taken are in the queues.

Copy the whole database. The record of applied migrations is part of it, and the new region must not apply them again.

§5. Deploy the new region

$ npx toa export secrets us
$ npx toa conceal ...
$ npx toa deploy us

It drains the queues and starts publishing its own changes. Changes the copy already contained arrive again and are ignored.

§What to watch

  • The depth of the new region’s queues between steps 3 and 5. They hold every change made in that time. Do not leave a long gap on a busy application.
  • Returned messages in the old region’s log. After step 3, one means a component has no queue on the other side: it is deployed in one region and not the other.
  • An error about a record of this region’s own rank. Two regions share a rank. Fix the declaration and redeploy.

§Adding a component later

A new component that stores state reaches one region before the others, and until every region has a queue for it, its changes are dropped there.

So before deploying a release that adds such a component, declare its queue on every region’s convergence broker, from a checkout that has the component:

$ npx toa export convergence eu | curl ... # the eu broker
$ npx toa export convergence us | curl ... # the us broker

The export lists the same components for every region, and the import adds only what is missing. Then deploy.

§Removing a region

Stop deploying to it, delete its federation upstream on every other region’s broker, and delete its queues. A queue left bound keeps filling with nothing reading it.

§Operating regions

  • A link that is down delays convergence and nothing else. Each region keeps serving from its own database, and changes cross when the link returns.
  • A halt reaches one region. Each region is a separate deployment; halting another means sending the signal to its gateway.
  • Realtime streams are written in every region a change reaches, so a client connected to any region sees changes made in all of them.
  • What to watch day to day is in Look After a Running Application.
Search
Search the documentation