Stop Everything for Maintenance

The broker needs an upgrade, the database a failover, or a release cannot run beside the previous one. For that the whole deployment has to stop — without cutting off an order half-approved. A halt does it: every process finishes what it has, closes every connection to the broker, the database and the cache, stays down for an interval you choose, and comes back by itself. Halt explains how and why it can be trusted.

§Turn it on

Halt is off by default. Turn it on in the Context:

# context.toa.yaml
introspection:
  halt: true

It is part of introspection, which in turn needs the gateway, so both must be in use. The application toa create wrote runs both, in sys.

With halt on, no component may declare introspection: false in its manifest. One that does fails the build.

§Limits

introspection:
  halt:
    duration: [30, 3600]
    quiescence: [30, 1800]

Each pair is [min, max] in seconds. These values are the defaults, and halt: true means exactly them.

• duration — how long the deployment may stay down.
• quiescence — how long it may be given to finish its work.

A request outside the limits is not refused. It is carried out with the nearest allowed value.

§Request a halt

A halt is an HTTP request to the gateway of the deployment being halted. There is no CLI command for it.

POST /introspection/signals/ HTTP/1.1
content-type: application/yaml

type: halt
seconds: 300
quiescence: 60

To read the limits a request must fit:

GET /introspection/signals/ HTTP/1.1

halt:
  duration: [30, 3600]
  quiescence: [30, 1800]

Both routes require the system:halt role; see Roles. A deployment with halt off has neither route.

§What you see

From the moment of the request the gateway answers 503, schedules stop firing, and the deployment has quiescence seconds to finish what it already had. Then one of two things happens.

The deployment was idle: it stops. Every connection closes, and after seconds every process comes back and carries on. Processes are not restarted or replaced, and they keep answering their readiness probe, so the cluster sees nothing wrong.

Something was still working: the halt is called off. Nothing was closed, and every process resumes at once. A deployment is never stopped while it is busy.

Nothing reports the outcome back to the request. It is in the signal records of introspection: a stop record names the halt it answers, so a halt with no stop beside it was called off.

§What clients see

• Requests to the gateway get 503 with a retry-after header saying when the deployment is back. Clients should honour it.
• Calls between components made just before the stop wait and are served when the deployment returns, within the caller’s own timeout.
• A reply that was streaming is cut when connections close. Its reader has to ask again.
• Pulses and schedules that would have fired during the halt are skipped, not made up afterwards (Run Work on a Schedule).
• Delayed calls that came due during the halt are made when the deployment is back, unless their own overdue setting says they are then too late (Do Something Later).

§What your components must do

For most components: nothing. Operations, receivers, schedules and everything else that goes through the operation context are stopped and restored for you.

What needs your attention is anything a component started itself in a run command (Keep Something in a Process): a timer, a file watcher, a connection to an external system. If it keeps working, the deployment never looks idle and every halt is called off.

Such a component stops and restarts its own things with the pause and resume run commands:

// components/store.catalog/rc/poller.ts
import type { Context } from '../types/index.d.ts'

export function preflight (context: Context) {
  context.state.poller = setInterval(() => poll(context), 1000)
}

export function pause (context: Context) {
  clearInterval(context.state.poller)
}

export const resume = preflight

pause runs when the halt is requested, while the component is still serving. Release there whatever would otherwise keep working.

What pause released has to be taken again, and a component comes back in one of two ways:

• The halt went through. The component is built anew, and preflight runs, as on any start.
• The halt was called off. The component is the same one, with the same context.state and everything it opened still open. Only resume runs.

So both preflight and resume must restore what pause stopped. Usually they are the same function, as above.

A component that exports pause without resume does not start.

context.state requires the component to declare state: ~. poll stands for whatever the timer does.

§Module scope survives a halt

Your modules stay loaded through a halt, so a variable at module scope is the same object afterwards. A connection kept there is neither closed by the halt nor reopened after it:

// kept at module scope: the halt does not touch it
const client = connect()

Either keep such things in context.state and manage them in preflight, pause and resume, or close and reopen them yourself in the same commands.

§Forgotten timers

A timer that was not stopped fires while the component is down and calls through a context that no longer exists. During the halt that call is refused and logged as a warning. Once the process is working again, the same stale call ends the process.

Test pause and resume with the stage; see Write a Test.

§Choosing the numbers

• seconds is how long the work you are doing takes, plus margin. A halt cannot be ended from outside. The deployment comes back when the interval is over, whether or not you are finished. Ask for an interval you can afford to be down for.
• quiescence is how long your slowest in-flight work takes. A deployment with long tasks needs a longer one. If halts keep being called off, either lengthen it or find the component that does not stop.

The one way to end a halt early is to start a new process: one that starts during a halt comes up working. Whether you can do that depends on what supervises your processes.

§Deploying a release that cannot be rolled

Some changes cannot be deployed by a rolling update, because old and new replicas cannot serve together: an input property that changes type, an entity property that changes meaning, a migration the previous release cannot read. See Changes that cannot be rolled.

1. Request a halt with an interval the rollout fits into.
2. When the deployment is down, deploy the new release: toa deploy production --wait.
3. The new processes start working immediately. The halted ones are replaced by the rollout and never serve beside them.

Since the halt waited for all work to finish, nothing in transit was shaped for the old release.

If the halt is called off, nothing happened: fix what kept the deployment busy, or ask for a longer quiescence, and try again. Do not deploy the incompatible release without it.

§Limits

• It covers the processes that were connected when it was requested. A process that starts while the deployment is finishing its work runs normally, counts as activity, and the halt is called off.
• It reaches one deployment. Regions are separate deployments; each is halted through its own gateway.
• A second request during a halt is ignored.
• It is for a healthy deployment. A process that is disconnected from the broker never hears the signal. A halt is not a way to bring a broken deployment to rest.

§Rehearse it

A halt is used rarely and under pressure, and one new component with an unmanaged timer is enough to break it. Run a halt on a schedule, with a short interval, where nothing is at stake. The day it is needed should not be the first time it is tried.