Process an Uploaded File

A product picture is uploaded: it has to be checked, attached to the product, resized. A workflow is a list of operations called once a file is stored, each with a reference to the file and not its bytes.

§A first workflow

pictures is the component Upload and Download Files writes: it names the storage pictures and exposes it. Replace its exposition with one that takes a picture of a product and runs a workflow:

# components/pictures/manifest.toa.yaml
exposition:
  /products/:product:
    octets:context: pictures
    io:output: true
    POST:
      octets:put:
        accept: image/*
        workflow:
          attach: pictures.attach

attach is the step’s name, and pictures.attach the operation it calls, written in full.

Each operation receives a reference to the file, not its bytes:

// components/pictures/operations/attach.ts
import type { AttachInput, Context } from '../types/index.d.ts'

export async function effect ({ path, entry, parameters }: AttachInput, context: Context) {
  if (entry.size < 1024) return new Error('TOO_SMALL')

  const reply = await context.remote.store.catalog.depict({
    query: { id: parameters.product },
    input: { picture: path }
  })

  if (reply instanceof Error) return new Error('NO_PRODUCT')

  return { picture: path }
}

An operation that needs the content reads it from the storage itself, with context.storages.

A workflow operation is an ordinary operation, called with this input and no query — so it is typically an effect or a computation that calls whatever has to change. Declare its input schema accordingly:

# components/pictures/manifest.toa.yaml
operations:
  attach:
    errors: [TOO_SMALL, NO_PRODUCT]
    input:
      type: object
      properties:
        storage: { type: string }
        path: { type: string }
        entry:
          type: object
          properties:
            size: { type: integer }
          required: [size]
        parameters:
          type: object
          properties:
            product: { type: string }
          required: [product]
        authority: { type: string }
        identity: { type: string }
        steps: { type: object }
      required: [storage, path, entry, parameters]

context.remote calls another component. store.catalog is the shop’s component of products, which are identified by a name such as kettle (Identifiers of another shape), and depict sets the picture of one. attach relies on this much of it:

# components/store.catalog/manifest.toa.yaml
name: catalog
namespace: store

entity:
  associated: true
  properties:
    id: { type: string, pattern: '^[a-z0-9-]+$' }
    picture: { type: string }

operations:
  depict:
    forward: transit
    concurrency: retry
    query: true
    input:
      type: object
      properties:
        picture: { type: string }
      required: [picture]

§Order

Steps written together in one object — a unit — run in parallel. A list of units runs in sequence:

        workflow:
          - check: pictures.moderate      # first
          - attach: pictures.attach        # then these two,
            index: store.search.index           # in parallel

pictures.moderate and store.search.index stand for two more operations that take the same input as attach.

A step that returns an error or throws stops the workflow: the steps running beside it complete, and no further unit starts.

§The reply

With a workflow the reply to an upload is multipart. The first part is the entry, sent as soon as the file is stored. One part follows for each step as it completes:

POST /pictures/products/kettle/ HTTP/1.1
content-type: image/jpeg
accept: application/yaml

HTTP/1.1 201 Created
content-type: multipart/yaml; boundary=cut

--cut
id: eecd837c9f2b4a1d8e7c6b5a4f3e2d1c
type: image/jpeg
size: 48211
checksum: 10cf16b458f759e0d617f2f3d83599ff
--cut
step: check
status: completed
--cut
step: attach
status: completed
output:
  picture: /pictures/products/kettle/eecd837c9f2b4a1d8e7c6b5a4f3e2d1c
--cut--

A step’s part holds:

A step that refuses is completed with an error:

step: attach
status: completed
error:
  code: TOO_SMALL

The status of the response is 201 in every case: the file was stored before the workflow began. A client reads the parts to learn how the workflow went. A refused step does not remove the file; an application that wants it gone deletes it, from the client or from an operation.

A step whose operation answers a stream sends a part with output for each value it yields, without status, and a last part with status: completed.

§One reply instead of parts

Most clients want one object: the entry, or what a step returned. Compose it with flow:compose, where $ is the list of parts:

    POST:
      octets:put:
        accept: image/*
        workflow:
          attach: pictures.attach
      flow:compose:
        id: $[0].id
        type: $[0].type
        picture: $[1].output.picture

HTTP/1.1 201 Created
content-type: application/yaml

id: eecd837c9f2b4a1d8e7c6b5a4f3e2d1c
type: image/jpeg
picture: /pictures/products/kettle/eecd837c9f2b4a1d8e7c6b5a4f3e2d1c

§Steps as tasks

A step whose endpoint is prefixed with task: is handed over as a task: it is accepted and runs on its own, after the request, with the retries a task has.

        workflow:
          - attach: pictures.attach
          - resize: task:pictures.resize

pictures.resize stands for an operation declared the way attach is. The step’s part is sent when the task is accepted, with output: null. Use tasks for processing that is slow or should survive the client leaving.

§When the client leaves

If the client disconnects, or the gateway is stopping, the step in flight completes and no further step starts. Steps that must run regardless are tasks.

§Workflows on delete

octets:delete may run a workflow before the entry is deleted:

    /*:
      DELETE:
        io:output: true
        octets:delete:
          workflow:
            detach: store.catalog.detach

store.catalog.detach stands for an operation that clears the product’s picture; it is given the same input as a step of an upload. The reply is 202 Accepted with the steps as parts. An error returned by a step prevents the deletion. The same holds if the client leaves before the workflow ends.

§Workflows without storing

octets:workflow runs a workflow on the file at the request path, and neither stores nor deletes anything:

    /*:
      DELETE:
        io:output: true
        octets:workflow:
          archive: pictures.archive

Here a DELETE archives a picture instead of deleting it; pictures.archive is the operation that would do it.

The reply is 202 Accepted with the steps as parts.

§What clients are told

A route whose upload starts a workflow says so in the answer to OPTIONS, as stream: true under octets: the reply arrives as parts, the first of which is the entry.

§Try it

The scenario uploads a picture, checks the composed reply of One reply instead of parts, and reads the file back through a /* route with octets:get. The whole exposition it runs against, open to everyone for the test:

# components/pictures/manifest.toa.yaml
exposition:
  /products/:product:
    anonymous: true
    octets:context: pictures
    io:output: true
    POST:
      octets:put:
        accept: image/*
        workflow:
          attach: pictures.attach
      flow:compose:
        id: $[0].id
        type: $[0].type
        picture: $[1].output.picture
    /*:
      GET:
        octets:get: ~

Sending a file takes a step the generated features/steps do not have. Add it beside them; it uses the agent and the World of features/steps/http.ts, and reads the file from features:

// features/steps/files.ts
import { createReadStream } from 'node:fs'
import { resolve } from 'node:path'
import { When } from '@cucumber/cucumber'
import type { World } from './http.ts'

When(
  'the file `{word}` is sent with the following request:',
  async function (this: World, file: string, head: string) {
    const path = resolve(import.meta.dirname, '..', file)

    await this.agent.stream(head, createReadStream(path))
  }
)

# features/pictures.feature
Feature: Product pictures

  Scenario: A picture is attached to its product
    When the file `kettle.jpg` is sent with the following request:
      """
      POST /pictures/products/kettle/ HTTP/1.1
      content-type: image/jpeg
      accept: application/yaml
      """
    Then the response is received:
      """
      201 Created

      id: ${{ picture.id }}
      type: image/jpeg
      """
    When the request is sent:
      """
      GET /pictures/products/kettle/${{ picture.id }} HTTP/1.1
      """
    Then the response is received:
      """
      200 OK
      content-type: image/jpeg
      """

A scenario starts the components itself and sends its requests to the gateway, which is part of sys:

$ npm run env
$ npm run sys
$ npm run features

Or with curl, reading the parts as they arrive. Without flow:compose on the route they are the parts of The reply:

$ npm run restart
$ curl -N http://localhost:8000/pictures/products/kettle/ 
    -H 'content-type: image/jpeg' -H 'accept: application/yaml' 
    --data-binary @kettle.jpg

Why the gateway moves the bytes and operations only decide is in Files.

§Summary