Follow a Request through the Application

Approving an order takes two seconds and nobody knows where they go: in store.orders, in the call to billing.wallets, in the database, or in the mail provider. A trace answers that. It is the record of one request as spans: one per step, each with its duration and its place among the others.

Spans for operations, calls, events and storage commands are made for you. You add spans around steps of your own, and say in the Context where traces go.

§See spans on the console

Add telemetry to the Context: name the console exporter and lower the log level to trace for the local environment:

# context.toa.yaml
telemetry:
  logs:
    level@local: trace
  traces:
    exporters@local:
      console: ~

Restart the application, call an operation — store.catalog.convert is the one Add a span writes — and read what the components wrote:

$ npm run restart
$ npx toa call store.catalog.convert "{ input: { amount: 1000, currency: 'EUR' } }"
$ pm2 logs app

Each span is written when it completes, as a log entry of TRACE severity whose message is the span’s name:

{
  "time": "2026-01-01T12:00:00.000Z",
  "severity": "TRACE",
  "message": "fetch rate",
  "attributes": { "currency": "EUR" },
  "context": { "namespace": "store", "component": "catalog", "operation": "convert" },
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_id": "53995c3f42cd8ad8",
  "duration": 12.4
}

These are log entries of the lowest severity, so they appear only when the log level is trace.

§What is traced without any code

A request that arrives with a W3C Trace Context (traceparent) continues that trace; otherwise a new trace starts.

Receivers handle an event after the operation that caused it has returned, so their spans may end later than the operation’s span.

A span is marked as failed (status: error) when an exception was thrown, when an operation answered with an exception, and when an HTTP request ended in a 5xx. A returned error is not a failure and leaves no mark.

Storage commands issued outside a recorded trace, such as index management at startup, are not recorded.

§Add a span

context.span runs a function and records how long it took. The catalog converts a price with a currency rate it asks an external service for:

# components/store.catalog/manifest.toa.yaml
name: catalog
namespace: store

operations:
  convert:
    input:
      type: object
      properties:
        amount: { type: integer }
        currency: { type: string }
      required: [amount, currency]

// components/store.catalog/operations/convert.ts
import type { Context, ConvertInput } from '../types/index.d.ts'

export async function computation (input: ConvertInput, context: Context) {
  const rate = await context.span('fetch rate', () => lookup(input.currency))

  return Math.round(input.amount * rate)
}

lookup stands for the request to the rate service.

context.span<T>(name: string, task: () => T | Promise<T>): Promise<T>
context.span<T>(name: string, attributes: object, task: () => T | Promise<T>): Promise<T>

• name is the span’s name. Keep it constant, as with log messages; put what varies into attributes.
• attributes is optional and is recorded with the span as given.
• task is the function to run. Its result is returned as it is. If it throws, the span is marked as failed and the exception is thrown on.

context.span always returns a promise, also for a synchronous task.

§Attributes

const rate = await context.span('fetch rate', { currency: input.currency }, () =>
  lookup(input.currency))

The rules for log attributes apply: identifiers and enumerable values, no personal data, no secrets.

§Nesting

Spans created inside a task are children of its span, and log entries written inside it carry its span_id:

export async function computation (input: ConvertInput, context: Context) {
  return context.span('convert', { currency: input.currency }, async () => {
    const rate = await context.span('fetch rate', () => lookup(input.currency))

    context.logs.info('Rate received', { rate })

    return Math.round(input.amount * rate)
  })
}

§When to add one

Calls to other components, storage, the stash and outside HTTP are already spans. Add one around what cannot be seen from outside your code: a computation that may be slow, or a call made with a client library of your own.

A span’s duration is recorded whether or not the trace is sampled, so a duration needs no metric of its own.

§Send traces to a backend

Trace ids are always there, in every log entry. Spans are recorded somewhere once the Context names an exporter:

# context.toa.yaml
telemetry:
  traces:
    exporters:
      otlp:
        endpoint: http://tempo:4318

Only the exporters named are used. To have both, name both:

telemetry:
  traces:
    exporters:
      console: ~
      otlp:
        endpoint: http://tempo:4318

The other options of otlp (timeout, cooldown, headers) and what happens when the backend is away are in Send Telemetry to Grafana.

Over OTLP, a span’s service.name is the component that produced it, such as store.orders, or exposition for the gateway’s request spans, also when several components share a process. There is no per-process service name to configure.

§Record fewer traces

Recording every trace of a busy application is expensive. Two settings limit what is recorded:

# context.toa.yaml
telemetry:
  traces:
    sample: 0.1
    rate: 5

sample keeps the recorded volume proportional to traffic; rate is a cap against spikes. With both set, a trace is recorded when it passes the sample draw and the rate is not exceeded.

A value out of range stops the deployment:

telemetry.traces.sample must be a number within [0, 1]
telemetry.traces.rate must be a positive number

What to expect with sampling on:

• A trace is recorded whole or not at all; there are no traces with components missing. Nothing needs to be configured for sampling in a collector.
• Operations of an unsampled trace run the same, and their log entries still carry its trace_id. A log record may therefore name a trace the trace backend never received.
• Metrics are unaffected: every operation is counted and timed whatever happens to its trace.

More on the subject is in the model article Telemetry.

§See where startup time goes

Set TOA_BOOT_TRACE=1. toa compose and toa serve then produce a trace of the boot: loading manifests, creating components, connecting each connector.

$ TOA_BOOT_TRACE=1 npx toa compose ./components/*

It is off by default.