Indexes and Data Migrations

The shop lists a customer’s orders newest first, and the list must stay fast as orders pile up. Order numbers must be unique. Deleted orders are purged after thirty days. Later, every order gets a note, and the old ones need a default. Each of these is a migration: a file in the component’s migrations directory, applied when the component starts.

§Indexes

# components/store.orders/migrations/0001-indexes.yaml
- index:
    name: index_customer
    keys: { customer: asc, CREATED: desc }

Restart; the index is made before the component serves. The query it is for:

$ npm run restart
$ npx toa call store.orders.enumerate "{ query: { criteria: 'customer==8d1f0c2e5a7b4c3d9e6f1a2b3c4d5e6f', sort: ['CREATED:desc'], limit: 20 } }"

A file is a list of steps, applied in order, written as YAML or JSON. The file name without its extension is the migration’s id, and migrations are applied in the order their ids sort — so number them:

migrations/
  0001-indexes.yaml
  0002-note-defaults.yaml

Declare an index for every query the application makes often: the properties its criteria filter on, followed by those it sorts by.

`index` field
`name`	The index’s name.
`keys`	Properties, each `asc`, `desc`, `hash` or `text`. A property inside an object is named by its path: `delivery.city: asc`.
`unique`	`true` to refuse a second object with the same keys.
`sparse`	`true` to index only objects that have the property.
`partial`	A filter; only matching objects are indexed.
`ttl`	Seconds after the moment in the key at which the object is removed.

A text index is what a query’s search uses.

To change an index, declare it again under the same name in a new migration: an index whose name is taken by one of another shape is dropped and made again. To remove one:

- dropIndex:
    name: index_customer

§A unique value

# components/store.orders/migrations/0002-unique-number.yaml
- index:
    name: unique_number
    keys: { number: asc }
    unique: true
    partial: { DELETED: null }

A second order with the same number is refused: the call fails with an exception — over HTTP, 409.

Always add partial: { DELETED: null } to a unique index. A deleted object stays in storage, and without the filter it keeps its value for good: the number could never be used again. The same filter keeps any index smaller.

§Expiry

A ttl index removes objects some time after a moment they hold. The system timestamps are moments, so purging deleted orders needs no property of its own:

# components/store.orders/migrations/0003-purge.yaml
- index:
    name: index_deleted
    keys: { DELETED: asc }
    ttl: 2592000 # thirty days after deletion

Live objects have DELETED: null and are never removed by it.

§Changing data

# components/store.orders/migrations/0004-note-defaults.yaml
- update:
    filter: { note: { $exists: false } }
    update: { $set: { note: '' } }

Step	Fields
`update`	`filter`, and `update` — an object, or a list to run as an aggregation pipeline.
`delete`	`filter`; `{}` means every object.
`images`	`true` to keep what an object was before each change, which reading only what changed needs; `false` to stop.

Steps are written in MongoDB’s own dialect. A pipeline can compute from existing values — which is how a property is converted when a moment format is added to it:

- update:
    filter: { expires: { $type: number } }
    update:
      - $set: { expires: { $toDate: { $toLong: '$expires' } } }

§Rules

Never edit a migration that has been applied; add a new one. A migration is applied once per database — not per replica, not per start — and never again. An index dropped by hand is not made again either; write another migration.

Write every migration so that it can be applied twice. One that was interrupted is applied again from its first step.

Keep it short. A migration runs before the component serves, and every replica waits for it. A long backfill over a large collection does not belong here: make it an operation that something calls.

Make sure the previous release can read what it leaves. During a rolling update the new release applies the migration while the old one is still serving from the same collection. If the old code cannot cope — a rename, a property that changes meaning — stop the deployment first; see Halt.

A prototype’s migrations come first. What a prototype of your own declares is applied ahead of the component’s own.

MongoDB only. A component that declares migrations for another storage does not start:

Component 'store.orders' declares migrations, which storage '@toa.io/storages.null' does not apply