Files

Entity state is structured data, validated against a schema and stored in a database. Images, documents, and video are a different kind of thing: large, opaque, and expensive to move. They do not belong in entities, and they should not pass through operations as values.

The platform handles them separately, in two parts: storages that keep binary objects, and gateway directives that move them between clients and storages.

§Storages

A storage keeps binary objects under paths. An application declares the storages it has in its Context, where each is given a provider: a cloud object store, a media service, a file system. A component names the storages it uses and reaches them through the context:

# pictures/manifest.toa.yaml
storages: [photos]

const entry = await context.storages.photos.head(path)

As with entity storage, the component does not know which provider stands behind the name.

§Entries

Each stored object has an entry: its size, its media type, a checksum of its content, when it was created, and attributes the application attaches. The entry is what the application works with. The bytes themselves are rarely read by business logic.

Two properties of entries are worth knowing.

The type is detected, not trusted. A storage identifies the type of an object from its content. What the uploader claimed is checked against that, and an object whose content contradicts its claim is refused. A file named as an image that is not one does not get in.

Identical content is stored once. Objects are kept by their checksum, so the same file uploaded many times occupies storage once, under as many entries as there were uploads.

§Uploads without operations

A route can accept an upload directly into a storage:

/photos/:customer:
  auth:id: customer
  octets:context: photos
  POST:
    octets:put:
      accept: [image/jpeg, image/png]
      limit: 10MiB

The gateway streams the request body into the storage as it arrives. No operation is involved, and the content is never held whole in memory. The route states what types and sizes it accepts, and the usual directives decide who may upload.

Downloads and deletions are declared the same way. Reading a stored file through a route streams it from the storage to the client.

§Workflows

Storing a file is usually the start of something: a thumbnail is generated, the content is analyzed, an entity is updated to refer to it. A route declares this as a workflow, a list of operations called once the object is stored:

POST:
  octets:put:
    accept: image/*
    workflow:
      - check: pictures.moderate
      - resize: pictures.resize
        register: customers.attach

Each operation receives a reference to the stored object — the storage, the path, and the entry — not the bytes. Steps listed together run in parallel, and units run in sequence. A step that returns an error stops the workflow, and the client is told which step refused.

A step can also be handed over as a task, for processing that should outlive the request.

This division keeps each part in its element. The gateway moves bytes. Operations make decisions about a file using its entry and, where they need the content, read it from the storage themselves.

§Variants

Processing often produces derivatives of an object: resized images, transcoded video. A storage keeps these as variants of the entry they derive from, so the original and everything made from it are addressed together.

§Files and streamed calls

A streamed call also carries bytes through the system, and the two are easy to confuse.

A file upload is for content that is to be kept. It is stored first, durably, and whatever happens next refers to it. A failure in processing loses nothing, since the object is already there and the processing can be repeated.

A streamed call is for content that only passes through an operation and is not kept by it.

Where content is both stored and processed, it is stored first and processed from the workflow.