Look After a Running Application

The shop is in production. Somebody now has to answer “what is running?”, “why is this pod not ready?”, “where did that receipt email go?” — and know what to do next. Start from the symptom. The answers assume the application was deployed with the supplied tooling, as Deploy describes.

§What is running

The deployment is one Helm release, named after the Context:

$ helm status store -n store
$ helm history store -n store

Workloads are named after what they run, and pods are labelled by composition, component and service:

$ kubectl get deployments -n store
composition-core
composition-media
extension-exposition-gateway
extension-configuration-values

$ kubectl get pods -n store -l toa/composition=core
$ kubectl get pods -n store -l toa/component-store-orders
$ kubectl logs -n store -l toa/component-store-orders --tail 100

The label of a component is the quickest way from “something is wrong with orders” to the pods that run it, whatever composition it is in.

§Which version is deployed

$ npx toa export tags production

prints the image every workload of production would run for the sources you have checked out. Compare it with what the cluster runs to see whether a checkout is what is deployed.

In the registry, the :<environment> tag of each image points at what was last deployed to that environment.

For a single component, the map mounted into every pod states the version of each component the deployment intends to run:

$ kubectl exec -n store <pod> -- zcat /etc/toa/.map.json.gz | grep store.orders

§A pod does not become ready

A pod is ready when all its components are connected to everything they use. One that stays unready is waiting for something: the broker, the database, a migration in progress.

$ kubectl describe pod -n store <pod>
$ kubectl logs -n store <pod>

• Search the log for Process failed. A process that cannot start, or fails later, exits and writes one such record with the reason — command failed, uncaught exception or unhandled rejection — and the error.

• A pod restarted after five minutes did not finish starting in time: a slow preflight run command or a long migration. See the limits.

• A version mismatch says so:

  'store.orders' is composed at a version the component map does not state. Run `toa map`.

  In a deployment this means an image and the map came from different sources, which happens when an evicted component is built separately.

• Is the infrastructure reachable from the cluster? Check from inside it:

  $ npx toa shell
  $ npx toa shell mongo
  $ npx toa shell -- nslookup rmq.internal.example.com

  toa shell starts a throwaway pod in the current Kubernetes context and opens a shell in it, with an image of your choice. The pod is removed when you leave.

• Is the secret what you think it is?

  $ npx toa reveal mongodb.default

§A message keeps failing

Symptom: the receipt email after an approval never arrives; a background task never completes; the log shows AMQP message discarded.

An operation that runs because of an event, or as a task, has nobody waiting for it. When it throws, the message is delivered again — five attempts over a little more than two minutes — and after the last one it is parked: set aside on the broker, where it stays until a person deals with it. A message that can never succeed — one that does not fit the operation’s contract, a task for an operation that does not exist, a call that went round in a circle — is parked on its first delivery. A parked message is never dropped and does not hold up the ones behind it. Delivery explains the behaviour; Handle Repeated Calls is the component’s side of it.

§Where to look

Two signs that a message was parked:

• the process logs an error, AMQP message discarded, with the queue, the operation and the exception message;
• the counter toa.amqp.diagnostics is incremented with condition="discard" (Metrics). Alert on it.

The messages themselves are in one queue, comq.parked, shared by every component and every application on that broker’s virtual host:

$ rabbitmqctl list_queues name messages | grep comq

A message there is as it was published, with headers saying what happened to it:

§What to do

1. Read x-comq-queue and x-comq-reason to find which operation failed and why.
2. Fix the cause: deploy the fix, restore the dependency, correct the data.
3. Put the message back on the queue x-comq-queue names, with the broker’s own tools, or remove it if the work is no longer wanted. Toa has no command for this.

Putting a message back is safe for an operation written to tolerate repeats, which every receiver and task has to be.

The queue grows until somebody drains it. Do not delete comq.parked or the comq.retry.* queues on a running system: the second kind holds the messages waiting for their next attempt.

If the cause is in the operation — it throws to refuse, where it should return an error — fix the operation: a returned error ends the work, a thrown one is retried and parked.

§Events are not published

Symptom: an order is approved and stored, but the event store.orders.approved reaches no receiver, or reaches an outside system late or never.

An event is stored together with the state change that caused it and published right after. One that could not be published stays stored and is tried again on a cycle until it goes out; it is never dropped and never expires. Transactional Outbox explains the guarantee, Announce a Change the declaration.

§Where to look

Was the event ever going to be published? A deployed component publishes an event only when a receiver in the application consumes it, or the Context names it in events. An event that another application reads and that is missing from events works locally and is silent when deployed.

What is stuck? Each component with events has a collection beside its entities, named <namespace>_<name>_outbox. What has not been published, oldest first:

db.store_orders_outbox.find({ published: false }).sort({ _id: 1 })

In a healthy deployment this finds nothing, or a few documents a moment old. Each document holds the event and, in outstanding, the destinations it has not reached. Documents that stay mean publication keeps failing:

Receivers must not assume that events arrive in order or only once; see React to an Event.

§Settings

# context.toa.yaml
outbox:
  interval: 5000   # milliseconds between checks for events left behind
  batch: 200       # how many of them one read brings back
  retention: 86400 # seconds a published change is kept before it expires

All three are optional.

• interval, at least 100, decides how fast a rare failure heals, not how fast events flow: recovery takes three to four intervals.
• batch, at least 1. A full read is followed by another, so it does not limit the work.
• retention, 0 or more. Published changes stay this long in the collection, as a change log you can read.

Two limits follow from how events are stored: a component runs at most 128 replicas, and a stored change holds the entity before and after it plus the operation’s input, so a 1 MB entity makes a document of about 2 MB against MongoDB’s limit of 16 MB.

To see how receivers cope with late events, three environment variables for development:

§Rolling back

There is no rollback command in Toa. Two ways:

• Deploy the previous sources. Check out the previous release and run toa deploy again. If its images are still in the registry, nothing is rebuilt.
• helm rollback store <revision>, which reapplies an earlier chart. It works as long as the images that chart pins still exist.

Either way, a rollback is a rollout in the other direction, and the rules of Release a New Version apply: the release you return to must be able to live with data the newer one wrote and with migrations it applied. Migrations are not undone.

How far back you can go is what your registry’s retention keeps; see Cleaning your registry.

§Changing things without a release

A secret. toa conceal updates it; restart the workloads that use it:

$ npx toa conceal mongodb.default password=n3w -n store
$ kubectl rollout restart deployment/composition-core -n store

Replicas and resources. They are declared in the Context. Change them there and deploy; when only they changed, nothing is rebuilt. A change made with kubectl scale lasts until the next deploy.

Configuration values. Declared in the Context and applied by a deploy. Changing Configuration Values says what is picked up without a restart.

§Who calls what

Two extension services publish a web page on the hosts of the Context’s ingress:

• the introspection explorer at /.introspection/: components, their operations and events, and which of them actually call each other;
• the configuration UI, for the values the configuration service serves.

The topology answers “who calls this?” before changing a contract, and “what would be affected?” before taking a component down. See Introspection and Changing Configuration Values. Logs, traces and metrics are in Logs, Traces and Metrics.

§Stopping for maintenance

To stop the whole deployment for a while — to upgrade the broker or the database, or to take a consistent backup — use a halt. It waits for work in progress, closes every connection, and brings the deployment back by itself.

Scaling workloads to zero also stops them, but cuts off whatever is in progress, which is then retried or parked as above. Prefer a halt when you can plan.

§A second copy on the same infrastructure

A preview, a load test or a rehearsal of an upgrade needs another copy of the application next to the first, on the same broker, database and cache, with the two seeing nothing of each other. Set TOA_SUFFIX; it is appended to the Context’s name in everything the processes keep:

$ TOA_SUFFIX=-preview npx toa compose ./components/*

• Letters, digits and hyphens only. Anything else, or an empty value, fails the start.
• The suffix is appended as is. Begin it with the separator you want.
• Context name plus suffix is at most 63 bytes. A longer one fails the start of a component with state.
• It is read when the process starts, from its environment or its .env file.
• A copy starts empty. It has its own database, queues and keys, and none of the first copy’s data.
• Not everything is separated. Retry and parked queues on the broker are shared. Files in storages and the brokers of regions are named as configured, so point a copy’s storages elsewhere if it must not touch the originals.
• Events from other applications do not arrive. A receiver with a source listens under the suffixed name, which no outside publisher uses. Test such flows without a suffix.

§Several regions

When the application runs as regions, also watch:

• the depth of the convergence.<namespace>.<name> queues on each region’s convergence broker. A queue that grows means a region is not applying changes;
• returned messages in a region’s log: a component exists in one region and not another;
• an error about a record carrying the region’s own rank: two regions share a priority.

§Before the application matters

• an alert on toa.amqp.diagnostics with condition="discard", and on the depth of comq.parked;
• an alert on unpublished outbox rows older than a few minutes;
• an alert on Process failed log records;
• registry retention that keeps what each environment tag points at;
• a scheduled halt rehearsal, if halt is something you count on;
• --wait on the deploy in the pipeline, so a failed rollout is noticed.