Keep a Log of What Happens and Query It

The shop wants to know what sells, what customers search for and where they leave the storefront, and wants to ask in words rather than build a dashboard. Chronicles keep a log of the application in a store the Context names: the changes of the components that name them, facts an operation sends, and what the storefront sends. An agent reads it over the gateway’s MCP. The store Toa provides is ClickHouse.

§Turn it on

Chronicles do two things: the service chronicles publishes every row of the log to a broker, and reads them back from a store. What takes the rows from the one into the other is the store’s setup, not the application’s: Toa ships it as reference SQL for ClickHouse, and a platform that runs many applications runs one broker and one ClickHouse for all of them.

Locally, the store is the ClickHouse beside the broker the application already has:

# docker-compose.yaml
services:
  clickhouse:
    image: clickhouse/clickhouse-server:26.8.34.13
    ports:
      - "8123:8123"
    environment:
      CLICKHOUSE_USER: developer
      CLICKHOUSE_PASSWORD: secret
      CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1

Then set it up as the reference says, once — the shared tables, with the engine that takes the rows off the broker, and the application’s user, named after the Context, which reads its rows and nothing else:

$ sed -e 's/{broker}/rabbitmq:5672/g' -e 's/{broker_username}/developer/g' -e 's/{broker_password}/secret/g' 
    node_modules/@toa.io/extensions.chronicles/store/clickhouse.sql 
  | docker compose exec -T clickhouse clickhouse-client -u developer --password secret --multiquery
$ sed -e 's/{scope}/shop/g' -e 's/{password}/secret/g' 
    node_modules/@toa.io/extensions.chronicles/store/clickhouse.scope.sql 
  | docker compose exec -T clickhouse clickhouse-client -u developer --password secret --multiquery

{broker} is the broker as ClickHouse reaches it — the compose service’s name and port — with its user and password; {scope} is the Context’s name, and {password} its user’s, secret as npm run env writes it.

Then the Context names both:

# context.toa.yaml
chronicles:
  broker: amqp://localhost:5672
  store:
    provider: clickhouse
    url: http://localhost:8123
chronicles@production:
  broker: amqp://analytics.internal
  store:
    provider: clickhouse
    url: http://analytics.internal:8123
  retention: 730 # days a row of events and of encounters is kept; absent, kept

The log is kept by a service of its own, chronicles, which takes the events the components declare off the broker they are published to, as any receiver does, and publishes them. Add it to what sys serves:

// ecosystem.config.js
{ ...common, name: 'sys', args: 'serve configuration exposition introspection chronicles' },

$ npm run env
$ docker compose up -d
$ npm run restart

npm run env writes the credentials into .env: the broker’s user, and the store’s user named after the Context. In a deployment they are a secret, which only the service is given:

$ npx toa conceal chronicles broker_username=... broker_password=... store_username=... store_password=...

While the broker of the log is away, nothing waits for it: what is to be published waits in the application’s own queues, and is published once it is back.

§Write without reading

Where the log is read elsewhere — by a platform that runs the application and reads the logs of all it runs — the Context names no store. The application then writes the log and does not read it: it has none of the routes and tools of Read the log, and its secret holds the broker’s user alone.

# context.toa.yaml
chronicles@production:
  broker: amqp://analytics.internal
  encounters: true

$ npx toa conceal chronicles broker_username=... broker_password=...

§Log the changes of a component

A component declares the events that are what happened to it, and the one that is the order as it stands — and of each, what the log keeps, with what every field means:

# components/store.orders/manifest.toa.yaml
chronicles:
  identity: customer
  events:
    created:
      description: An order was placed
      properties:
        total:
          type: integer
          description: The amount of the order, in cents
        items:
          type: array
          description: What was ordered
          items:
            type: object
            properties:
              sku:
                type: string
                description: The code of the product
              quantity:
                type: integer
                description: Units of the product
  state:
    event: snapshot
    description: An order as it stands
    properties:
      status:
        type: string
        enum: [new, paid, shipped]
        description: Where the order is in its life
      total:
        type: integer
        description: The amount of the order, in cents

created is an event every component has, and a row of the log every time an order is placed. snapshot is an event of the order’s own, which nothing else has to consume, and the log keeps the latest of it for every order:

// components/store.orders/events/snapshot.ts
import type { Event } from '../types/index.d.ts'

export function payload (event: Event) {
  return event.state
}

What is written is what the declaration names, at every depth, and nothing else: the customer’s email and address, which the order has, never reach the store, and neither does a field the order gains later until it is declared. So the declaration is where personal data is kept out. A payload whose declared field is of another type — a total that is a string — is not logged, and waits in the parking queue of the service.

Every field has a type and a description, at every depth, or the component is refused when it is read. A field is a string, an integer, a number, a boolean, an object of fields or an array of any of these; a string may name an enum or a format — date-time, date, uuid, ipv4 — which is the type it is read as. The descriptions are what an agent reads to know that total is in cents.

Which order changed and its version are read from the payload’s id and VERSION, before the declaration: a state must carry both, or it is refused, and neither is declared. When it changed is the payload’s UPDATED, which the order as it stands carries, or else the moment the component made the change; timestamp: placed names another property of the payloads that says when, and timestamp: ~ none. A change the service takes late is still logged at its own time. identity names the order’s property that says whose it is, which is the identity column of its rows; absent, it is identity, and identity: ~ is none. A change is logged however long the service is away, and once however often it is delivered; one committed before the service first ran is not.

§Send a fact

A search changes nothing, and is still worth knowing:

// components/store.search/operations/find.ts
import type { Context, FindInput } from '../types/index.d.ts'

export async function computation (input: FindInput, context: Context) {
  const results = await query(input.text, context)

  await context.chronicles.send('searched', { text: input.text, results: results.length })

  return results
}

# components/store.search/manifest.toa.yaml
chronicles:
  facts:
    searched:
      description: A customer searched the catalogue
      properties:
        text:
          type: string
          description: What was typed
        results:
          type: integer
          description: How many products were found

A fact is declared as an event is, and send throws for a name the component does not declare or a field of another type. It resolves once the broker has the fact, which then waits for the service as an event does. An operation that knows whose the search is passes it as a third argument, { identity: input.customer }: the service is not told who called the operation.

§Take encounters from the storefront

What the storefront tells of its visitors — a page viewed, a button pressed — is an encounter:

# context.toa.yaml
chronicles:
  encounters: true

The storefront keeps an id of the visitor, in localStorage, and one of the session, and posts batches of what happened, at most 100 at a time. A signed-in customer’s go to the route of their identity, with their token, and are that customer’s rows:

await fetch(`https://api.store.example/chronicles/encounters/${customer.id}/`, {
  method: 'POST',
  keepalive: true,
  headers: { authorization: `Token ${token}`, 'content-type': 'application/json' },
  body: JSON.stringify({
    sent: Date.now(), visitor, session,
    events: [{ id: crypto.randomUUID(), name: 'page_viewed', time: Date.now(), payload: { path: location.pathname } }]
  })
})

A visitor’s go to the route of no one, which takes no credentials, so navigator.sendBeacon can send them as the page closes, as a form whose events is their JSON:

navigator.sendBeacon('https://api.store.example/chronicles/encounters/', new URLSearchParams({
  sent: Date.now(), visitor, session,
  events: JSON.stringify([{ id: crypto.randomUUID(), name: 'page_left', time: Date.now(), payload: { path: location.pathname } }])
}))

202 Accepted

accepted: 1

visitor and session are the storefront’s own, at most 64 characters each, and either may be left out. The same visitor before signing in and after is what tells which visits became which customer: the rows of the route of an identity have both. A visitor is not a person — a customer may have many, and a browser that clears its storage is a new one.

A batch says when it is sent, by the browser’s clock. An event is { id, name, time, payload }: its own id, which a retry sends again; lowercase letters, digits and underscores; when it happened, by the same clock; an object of at most 16 KiB. Who may send to which route is in Who may do what.

A payload is logged as it was sent, whatever its shape: a storefront that changes what an event carries still has old pages open for months, so a shape that changes is a new name, checkout_started_v2, rather than a new meaning of the old one.

A browser’s clock may be off by anything; what it measures between two of its moments is not. So an encounter’s time is when the gateway received the batch, received, less how long before sent the event happened: the gateway’s clock, at the distance the browser measured. A batch a page kept while offline is logged at when its events happened.

Every encounter also has the ip and the country of its request, as the gateway reads them from the headers exposition.ip and exposition.country name. An address is personal data, and the log erases nothing, so it keeps the network the address is in — a /24 of IPv4, a /48 of IPv6 — unless the Context says chronicles.ip: full.

Every encounter also has the origin of the page that sent it and the browser’s agent, from the request’s Origin and User-Agent headers, and what the agent says the client is, by name and without versions: browser — Chrome, Safari, Edge, Firefox, Opera, Samsung Internet; os — Windows, macOS, iOS, Android, ChromeOS, Linux; and device — desktop, mobile, tablet, or bot for a crawler or a program, which a question of who visits leaves out. A body cannot say any of them for the request. Safari on an iPad says it is a Mac, and is counted as one.

§Document what it sends

What each encounter is, and what its fields mean, is the configuration of chronicles.encounters:

# context.toa.yaml
configuration:
  chronicles.encounters:
    page_viewed:
      description: A page was shown
      fields:
        path: The URL path, without the query
    page_left:
      description: The visitor closed the page or went elsewhere
      fields:
        path: The URL path of the page left

It is what an agent reading the log is told of each encounter, and what whoever writes the storefront reads, at GET /chronicles/encounters/. It documents, and checks nothing: an encounter it does not name is logged as any other.

§Read the log

The log is three tables. Every row has scope, the application it is of, and an application reads its own.

events holds a row for everything a component did. The order placed and the search made above are two rows of it:

A change is a row under <component>.<event> for every event it renders, and names the order that changed and its version. A fact is a row under <component>.<name>, and names none. Every row also has time, when it happened — never when it was written — and id, the same however often it is delivered.

states holds every order as it stands: the snapshot of its latest version, by component and entity. It is the order now and nothing before it: what an order was yesterday is read from its events, so an event that a question of history depends on has to carry what the question asks.

encounters holds what the storefront sent: name, time, received, visitor, session, identity, ip, country, origin, agent, browser, os, device and payload.

Of every declared field, the answer to GET /chronicles/queries/ says its type, what it means, and how to read it in that type:

SELECT toDate(time) AS day, count() AS orders,
  sum(JSONExtract(ifNull(toJSONString(payload.total), 'null'), 'Nullable(Int64)')) / 100 AS revenue
FROM events FINAL WHERE label = 'store.orders.created' GROUP BY day ORDER BY day

A value that is not of its type reads as NULL. A query is written in the store’s dialect. In ClickHouse, the tables are read with FINAL, which collapses what was delivered twice:

SELECT name, count() FROM encounters FINAL WHERE device != 'bot' GROUP BY name

SELECT visitor, groupUniqArray(identity) FROM encounters FINAL
WHERE identity IS NOT NULL GROUP BY visitor

Nothing arrives in order: order by time, or by version within one order.

§Read it over HTTP

The log is read through these routes of the gateway:

Who may call each is in Who may do what.

GET /chronicles/queries/ tells of every declared label and state what it means, its view, how many rows it has, the first and the last time of them — which tells whether rows still arrive — and its fields, with their types and descriptions. Of every encounter it tells the fields the latest payloads had, with the types they were sent as — two types of one field are two versions of the storefront — and what the configuration says of them.

A query reads the log and nothing else, and cannot write: for at most 30 seconds, 10,000 rows and 64 MiB, with at most 4 queries at once. A view is a statement a query could run; give it a title and a description, which tell an agent that finds it what it counts. With refresh, in seconds and at least an hour, it is computed on that cycle, read as a table, and told with when it was last refreshed. A store has at most 50 views. A refusal is a 422 with its code: REFUSED, with what the store said in cause.message, for a statement; RESERVED, for the name of a table or of a declared view, EXHAUSTED and NOT_FOUND for a view.

§Try it

POST /chronicles/queries/ HTTP/1.1
host: api.store.example
authorization: Basic cm9vdDpzZWNyZXQjMTIzNA==
accept: application/yaml
content-type: application/yaml

sql: SELECT status, count() AS n FROM store_orders GROUP BY status

201 Created

columns:
  - name: status
    type: LowCardinality(Nullable(String))
  - name: n
    type: UInt64
rows:
  - [new, 3]
  - [paid, 9]
truncated: false

§Let an agent ask

Publish the gateway’s tools, and each of these routes is a tool an agent holding its role is offered — chronicles.queries.GET, chronicles.queries.POST, chronicles.queries.views._name.PUT, chronicles.queries.views._name.DELETE and chronicles.encounters.GET:

# context.toa.yaml
exposition:
  mcp:
    name: Store

An agent asks chronicles.queries.GET what the log holds and how to read it — a guide Toa writes, for the log and for its store — then writes its queries and the views it keeps. What is particular to the application is in what it declares and documents, and in exposition.mcp.instructions.

§Who may do what

A caller without what a route takes is answered 403, and past its limit 429; the limit per address takes exposition.ip to name the address. An agent is offered the tool of a route only where it holds the route’s role.

The roles are what to grant whom — see Give Staff Wider Access:

• system:chronicles:query reads every row of the log, with the customers’ identities, networks, origins and browsers in it. It is access to personal data, and is granted as that.
• system:chronicles:views defines and drops the views every reader reads. A view reads only what a query could, and changes nothing of the log, but it changes what other readers are told a view counts.
• system:chronicles:encounters reads what the encounters are, and no row: it is for whoever writes the storefront.
• system:chronicles is all three.

The storefront sends events and nothing that says who or where the visitor is. identity is the route’s, whose credentials have to be the identity’s; ip, country, origin and agent are what the gateway read off the request, and a body cannot say any of them. Anything else in a payload is what the browser said — a user there is not a user — so a question of who sent an encounter asks identity.

A query runs as the application’s user of the store, which the store’s setup lets read the rows of its own scope, keep views in a database named after it, and lift none of its limits; Toa asks a query to be read-only besides. A user the setup gives no row policy reads every scope — the platform’s, which reads the applications it runs.