Count Things and Alert on Them

The business wants to see approvals per hour by sales channel, and to be alerted when they drop to zero. That is a metric: a number about the application as a whole, collected at an interval. A component declares the metrics that are about the business and records them through context.metrics. Durations, error counts and the like are already recorded for every application and are listed at the end.

§Count approvals

approve is the operation of store.orders that Create a Component writes. Declare the metric in the component’s manifest:

# components/store.orders/manifest.toa.yaml
telemetry:
  metrics:
    approvals:
      type: counter

and record it in the operation, which takes the context as one more parameter:

// components/store.orders/operations/approve.ts
import type { Context, State } from '../types/index.d.ts'

export async function transition (input: null, entry: State, context: Context) {
  if (entry.status !== 'pending')
    return new Error('NOT_PENDING')

  entry.status = 'approved'

  context.metrics.approvals.add(1)

  return entry
}

In the backend the series is named after the component and the declared name: store.orders.approvals. Two components may declare the same name without their series merging.

§Turn metrics on

Nothing is measured until the Context configures metrics, and nothing leaves a process until an exporter is named:

# context.toa.yaml
telemetry:
  metrics:
    exporters:
      otlp:
        endpoint: http://prometheus:9090/api/v1/otlp

The other options of otlp are in Send Telemetry to Grafana.

Without this annotation context.metrics still works, so the code is the same everywhere. It just records nothing.

§See it locally

There is no console exporter for metrics; reading them takes a backend. Run Prometheus locally with its OTLP receiver (the local stack), point the local environment at it, restart the application and approve a pending order by its id:

# context.toa.yaml
telemetry:
  metrics:
    interval@local: 1000
    exporters:
      otlp@local:
        endpoint: http://localhost:9090/api/v1/otlp

$ npm run restart
$ npx toa call store.orders.approve "{ query: { id: '7d8e…' } }"

The series appears in Prometheus after the next collection. An alert is a rule of the backend over that series: for a counter, on its rate.

§Split a metric with labels

A label splits a metric into series. Each label is declared with the values it admits. The channel has to reach the operation, so approve gets an input:

# components/store.orders/manifest.toa.yaml
telemetry:
  metrics:
    approvals:
      type: counter
      labels:
        channel: [web, app, partner]

operations:
  approve:
    concurrency: retry
    query: true
    errors: [NOT_PENDING]
    input:
      type: object
      properties:
        channel: { type: string }
      required: [channel]

The type of the operation’s first parameter is then ApproveInput, generated from that declaration, instead of null:

context.metrics.approvals.add(1, { channel: input.channel })

Every declared label takes a value on every call.

A value the label does not list is recorded as UNDECLARED. The operation is not affected, and the first such value for a label is written to the log:

// input.channel is 'kiosk', which the manifest does not list
context.metrics.approvals.add(1, { channel: input.channel }) // → { channel: 'UNDECLARED' }

A label declared with ~ admits any value:

labels:
  channel: [web, app, partner]
  warehouse: ~

Keeping the number of its values small is then up to you. Never put an identifier of an order, a customer or a request into a label: each distinct value is a series, kept for the whole retention of the backend.

labels: ~, or no labels at all, declares a metric with no labels.

§Measure a distribution or a level

# components/store.orders/manifest.toa.yaml
telemetry:
  metrics:
    approvals:
      type: counter
      labels:
        channel: [web, app, partner]
    totals:
      type: histogram
      unit: EUR
      buckets: [5, 20, 50, 100, 500]
      labels:
        channel: [web, app, partner]
    backlog:
      type: gauge

context.metrics.approvals.add(1, { channel: input.channel })
context.metrics.totals.record(entry.total / 100, { channel: input.channel })
context.metrics.backlog.set(pending.length)

A counter’s amount is positive. Choosing between the types:

• Something that can also go down is a gauge, never a counter.
• Do not count with a gauge: two changes within one interval export as one number, and a level that rose and fell between two collections is never seen.
• An outcome with a handful of values is a counter with a label, not a histogram.
• A duration needs no metric: context.span measures its task, and the duration is recorded whether or not the trace is sampled.
• An error code an operation declares needs no metric either: toa.operation.errors counts them per code.

§Declaration reference

Names of metrics and labels start with a letter, continue with letters, digits and underscores, and do not end with an underscore.

A malformed declaration is refused when the manifest is read, before anything starts: a name that is not an identifier, an unknown type, a histogram without buckets or with buckets that do not ascend, buckets on a metric that is not a histogram, or a label that lists UNDECLARED among its values.

Histogram 'totals' declares no buckets
Buckets of 'totals' must ascend

§An undeclared metric

Reading a name from context.metrics that the manifest does not declare throws:

Metric 'aprovals' is not declared

It is a defect in the code, and the exception is a permanent one: where the operation runs for an event or a task, the message is set aside on its first delivery instead of being tried again. See Refuse a Request.

§Choose intervals, buckets and labels

• Nothing shorter than interval is visible. A rate computed over a window shorter than about four intervals has too few samples to trust. Counters and histograms lose nothing between collections; a gauge does.
• Nothing finer than a bucket edge is visible. Choose edges around the values you care to tell apart. Changing the edges later starts the history over.
• A backend charges for series, not for measurements. Every combination of label values is a series, and every histogram bucket is a series of its own: twelve edges are fifteen series per combination. The number of series a metric can produce is the product of its label lists.
• interval multiplies the data points of every series; raise it where a backend bills by data point.

Counters and histograms are cumulative from the start of the process, and every series carries service.instance.id, so replicas do not collide and a lost export costs resolution, not totals.

Every series also carries the application’s name as service.namespace and the environment as deployment.environment.name. Several applications and environments can share one backend and are told apart by those, not by metric names.

§What is recorded for you

These are recorded for every application once metrics are on. Durations are in seconds. They are prefixed toa; a component’s own metrics are named after the component.

§Operations and calls

toa.operation.* is an invocation as the component serving it sees it. toa.call.* is the same invocation as its caller sees it, transport included; component there is the callee.

The three outcomes of an operation:

• an error it returned is toa.operation.errors, by the code it declares;
• an exception is toa.operation.exceptions, with outcome permanent or transient, depending on whether another attempt could succeed;
• success is the count of toa.operation.duration less the two, since the histogram observes every invocation.

toa.call.exceptions counts only failures of transmission, never an exception the callee sent back, which is counted where it happened.

§Storage, broker, events

toa.storage.conflicts counts concurrent changes to the same entity. Where the operation declares concurrency: retry the change is retried and succeeds, and this counter is the only place the conflict shows; otherwise it is also an exception of the operation.

toa.outbox.age is how long the oldest unpublished event has waited, and is the number to alert on for event delivery. toa.outbox.unassigned rising means this replica recovers nothing, which points at atomicity being unreachable. What to do about either is in Operating.

condition of toa.amqp.diagnostics is one of return, discard, taken, remove, lost, recover, flow, drain, close, error, reconnect, exhausted, open.

§Extensions

route of the gateway’s metrics is the route template, and none where a request matched no route.

§Process

Recorded by every process, the gateway included. toa.process.loop.delay covers the whole interval, not only the moment it was read: a high p99 means operations of that process are blocking the event loop.