Send Telemetry to Grafana

The operations team runs Grafana and wants the shop’s logs, traces and metrics there. All three leave the application as OTLP over HTTP, to whatever receivers the telemetry annotation of the Context names. Nothing changes in the components.

§Wire the application to a local stack

Add telemetry to the Context and restart the application:

# context.toa.yaml
telemetry:
  traces:
    exporters:
      otlp:
        endpoint: http://localhost:4318
  metrics:
    exporters:
      otlp:
        endpoint: http://localhost:9090/api/v1/otlp
  logs:
    exporters:
      otlp:
        endpoint: http://localhost:3100/otlp

$ npm run restart

Make a few requests, then look in Grafana: the entries are in Loki, the traces in Tempo, and toa.operation.duration in Prometheus.

In production, point each endpoint at any receiver of that signal and use headers for authentication, for a hosted service for example.

§What the stack needs

• Tempo with OTLP/HTTP enabled (port 4318).
• Prometheus started with --web.enable-otlp-receiver, which is where the application posts its metrics. If Tempo’s metrics-generator is used, also --web.enable-remote-write-receiver, which is where the generator writes.
• Loki with the OTLP receiver (/otlp/v1/logs), on the tsdb store with schema v13. Attributes of a record arrive as structured metadata, which only that schema allows.
• Grafana with a Prometheus datasource, a Loki datasource, and a Tempo datasource whose service-map setting points at the Prometheus one.

§Service graph

For Tempo’s service graph, enable its metrics-generator with the service-graphs and span-metrics processors, and add messaging.destination.name to peer_attributes. Events then appear as nodes between the component that publishes and the components that receive.

§From a trace to its logs

Every log entry written within a span carries its trace_id and span_id, so traces and logs link with two datasource settings.

Over OTLP a record’s line is the message alone; Loki keeps service.name as the service_name label and the trace and span ids as trace_id and span_id beside the line.

• Tempo datasource, Trace to logs: select the Loki datasource, set time shifts such as -5m and +5m, and a custom query, since the trace id is not in the line:

  {service_name="$SERVICE"} | trace_id = `${__span.traceId}`

  Append | span_id = `${__span.spanId}` to narrow it to one span.

• Loki datasource, Derived fields: matcher type Label, regex trace_id, query ${__value.raw}, internal link to the Tempo datasource.

Where records are instead collected from the processes’ console output, the line is the whole JSON entry. Use | json | trace_id = ... in the query, and a derived field matching the regular expression "trace_id":"([0-9a-f]+)".

Log entries of unsampled traces carry a trace_id that does not exist in Tempo; a link from such an entry finds nothing.

§Sampling

Sampling is set in the Context with sample and rate. No tail sampling is needed in a collector.

§Avoid duplicates

The console exporter for logs stays on unless logs.exporters.console: false. Turn it off where the console output of every process is also collected into the same Loki, or every record arrives twice.

§The whole annotation

# context.toa.yaml
telemetry:
  logs:
    level: info
    exporters:
      console: false
      otlp:
        endpoint: http://loki:3100/otlp
    store.orders:
      level: debug
  traces:
    sample: 0.1
    rate: 5
    exporters:
      otlp:
        endpoint: http://tempo:4318
  metrics:
    interval: 15000
    exporters:
      otlp:
        endpoint: http://prometheus:9090/api/v1/otlp
  ready:
    port: 8001

Every part is optional. With no telemetry annotation at all, an application writes log entries of info and above to the console, records no traces and measures no metrics.

§The otlp exporter

The three signals take the same options:

otlp:
  endpoint: http://collector:4318
  timeout: 5000
  cooldown: 30000
  headers:
    authorization: Basic ...

An otlp block without an endpoint stops the deployment:

telemetry.logs.exporters.otlp.endpoint is required

One receiver can take all three signals; give each section the same endpoint. YAML anchors keep that in one place:

telemetry:
  logs:
    exporters:
      otlp: &collector
        endpoint: http://collector:4318
  traces:
    exporters:
      otlp: *collector
  metrics:
    exporters:
      otlp: *collector

§Per environment

Usually only some environments have a backend. Suffix the exporter:

telemetry:
  logs:
    level: debug
    level@production: info
    exporters:
      console@production: false
      otlp@production:
        endpoint: http://collector.telemetry:4318
  traces:
    sample@production: 0.1
    rate@production: 5
    exporters:
      otlp@local:
        endpoint: http://localhost:4318
      otlp@production:
        endpoint: http://collector.telemetry:4318
  metrics:
    interval@production: 60000
    exporters:
      otlp@production:
        endpoint: http://collector.telemetry:4318

§When data is sent, and when it is lost

• Log records and spans are sent in batches: every 512 of them or every 5 seconds, whichever comes first. Metrics are sent every interval.
• Everything pending is sent when a process exits normally. A process killed outright (SIGKILL) may lose its last batch.
• An unavailable backend never slows the application or delays its shutdown. A failed batch is dropped, not queued; nothing is sent for cooldown, then the exporter tries again. One warning is logged when an outage begins and one entry when the endpoint recovers. A process shutting down waits at most one timeout for an unreachable endpoint.
• So an outage of the backend leaves a gap in logs and traces. Metrics are cumulative: the next export carries what the lost one would have.

§How data is labelled

Do not set OTEL_SERVICE_NAME or similar per process: spans are attributed to components, also when several components share a process.

§Readiness

Every composition answers GET /.ready on port 8001: 503 until the composition has connected to everything it needs, 200 after. A deployment uses it as the startup and readiness probe without anything being declared.

# context.toa.yaml
telemetry:
  ready:
    port: 8001
    path: /.ready

ready: false turns the probe off, and the deployment is then rendered without one.

When ready, a process also sends ready to its parent process (process.send('ready')), which process managers such as PM2 can wait for (wait_ready), as the ecosystem.config.js that toa create wrote does.

When several processes run on one machine, as in a local run, only the first gets the port. The others log Ready probe port already in use, skipping and serve no probe; they still send the ready signal.

The gateway answers a readiness probe of its own, on port 8004.