Keep Files in a Storage

A product has pictures, an order has an invoice in PDF. Images, documents and video do not belong in entities. An application keeps them in storages: named places for binary objects, declared in the Context and backed by a provider — a cloud object store, a media service, a directory.

A storage is declared, and then used from an operation. Moving files between clients and storages over HTTP needs no operation at all; see Upload and Download Files.

§Declaring a storage

The Context names each storage and gives it a provider. toa create wrote one, tmp; add pictures beside it:

# context.toa.yaml
storages:
  tmp:
    provider: tmp
    directory: store
  pictures:
    provider: fs
    path: /var/store/pictures

A component names the storages it uses. This one is new, and the manifest is all of it:

# components/pictures/manifest.toa.yaml
name: pictures

storages: [pictures]

operations:
  describe:
    input:
      type: object
      properties:
        path: { type: string }
      required: [path]
    errors: [NO_PICTURE]

and reaches them through the operation context:

// components/pictures/operations/describe.ts
import type { Context, DescribeInput } from '../types/index.d.ts'

export async function computation ({ path }: DescribeInput, context: Context) {
  const entry = await context.storages.pictures.head(path)

  if (entry instanceof Error) return new Error('NO_PICTURE')

  return { type: entry.type, size: entry.size }
}

pictures is the component of the shop that deals with product pictures.

A storage name is lowercase letters, digits and underscores, up to 32 characters. A component that names a storage the Context does not declare fails the deployment: Missing 'pictures' storage annotation declared in 'default.pictures' — default stands for the namespace of a component that has none.

A component that does not know the names in advance — a generic one, reused between applications — declares storages: ~.

A component whose routes move files in and out of a storage declares that storage too, whether or not its operations touch it.

§Entries

A stored object has an entry, which is what an application works with:

id: eecd837c
type: image/jpeg
size: 48211
checksum: 10cf16b458f759e0d617f2f3d83599ff
created: 2026-03-01T10:12:44.071Z
attributes:
  product: kettle

An object lives under a path: /products/kettle/eecd837c. The directory part is the application’s to choose; the last segment is the entry’s id.

Identical content is stored once: the same file uploaded a hundred times occupies the storage once, under a hundred entries. The type is detected from the content, not trusted from the uploader; see Putting an object.

§The storage interface

context.storages.<name> has four methods. Each returns its result or an Error: check with instanceof Error, as for any reply.

§Putting an object

const entry = await context.storages.pictures.put('/products/kettle', stream, options)

path is the directory; the entry gets a generated id and is stored at path/id. stream is a Readable. The result is the entry.

The type of the content is detected from its leading bytes for these types: image/jpeg, image/png, image/gif, image/webp, image/heic, image/jxl, image/avif.

• If the content is detected as one type and claim says another, the result is the error TYPE_MISMATCH. A file named as a JPEG that is a PNG does not get in.
• If the content cannot be identified and claim is not one of the known types, the claim is taken as the type. This is how an SVG or a PDF is stored.
• If accept is given and the type matches none of it, the result is the error NOT_ACCEPTABLE.

// components/pictures/operations/mirror.ts
import { Readable } from 'node:stream'
import type { Context, MirrorInput } from '../types/index.d.ts'

export async function effect ({ product, url }: MirrorInput, context: Context) {
  const response = await fetch(url)

  if (response.body === null) return new Error('NOT_A_PICTURE')

  const entry = await context.storages.pictures.put(
    `/products/${product}`,
    Readable.fromWeb(response.body),
    { accept: 'image/*', attributes: { product } }
  )

  if (entry instanceof Error) return new Error('NOT_A_PICTURE')

  return { id: entry.id }
}

§Reading an entry

const entry = await context.storages.pictures.head('/products/kettle/eecd837c')

Returns the entry, or the error NOT_FOUND.

§Reading the content

const object = await context.storages.pictures.get('/products/kettle/eecd837c')

if (object instanceof Error) return new Error('NO_PICTURE')

for await (const chunk of object.stream) { /* ... */ }

Returns the entry’s metadata with a stream of the content, or the error NOT_FOUND.

The path may address a variant of the object — a derivative kept with it — by extension: /products/kettle/eecd837c.thumbnail.jpeg. Which variants exist depends on the provider; see Cloudinary.

§An address to read it from

A file is handed to someone who reads it straight from the provider — a model, a browser, another service — by an address that needs no credentials of theirs:

const url = await context.storages.manuals.url('/products/kettle/eecd837c', { expires: 86400 })

if (url instanceof Error) return new Error('NO_MANUAL')

NOT_FOUND is a file an s3 or spaces storage does not have. An address signed for a while stops working once it expires: hand it to whatever reads the file now, and make another for whatever reads it later.

§Deleting

await context.storages.pictures.delete('/products/kettle/eecd837c')

Deletes the entry.

§Providers

The provider is chosen in the Context, per storage. Operations and routes do not change when it does.

§File system

storages:
  pictures:
    provider: fs
    path: /var/store/pictures

In a deployment the directory can be a Kubernetes persistent volume claim, mounted at path:

storages:
  pictures:
    provider: fs
    path: /var/store/pictures
    claim: pictures-pvc

§Temporary directory

A directory under the operating system’s temporary directory. For local development and tests:

storages:
  pictures:
    provider: tmp
    directory: store-pictures

§Amazon S3

storages:
  pictures:
    provider: s3
    bucket: store-pictures
    region: eu-west-1

Secrets ACCESS_KEY_ID and SECRET_ACCESS_KEY are optional. Without them the standard AWS credential chain is used: environment variables, the shared configuration file, the instance’s role.

§DigitalOcean Spaces

storages:
  pictures:
    provider: spaces
    space: store-pictures
    region: fra1

region is the datacenter’s slug. Secrets ACCESS_KEY_ID and SECRET_ACCESS_KEY are required.

§Cloudinary

Cloudinary stores media and transforms it on request.

storages:
  pictures:
    provider: cloudinary
    environment: store-cloud
    type: image
    prefix: store/pictures

Secrets API_KEY and API_SECRET are required.

Variants. An object is requested transformed by appending dot-separated extensions to its path: /products/kettle/eecd837c.200x200.webp. Each entry of transformations declares one kind of extension as a regular expression, and the Cloudinary transformation it stands for. Named groups of the expression fill the <placeholders>:

storages:
  pictures:
    provider: cloudinary
    environment: store-cloud
    type: image
    transformations:
      - extension: (?<width>d*)x(?<height>d*)(z(?<zoom>d*))?
        transformation:
          width: <width>
          height: <height>
          zoom: <zoom>
          crop: thumb
          gravity: face
        optional: true
      - extension: (?<format>jpeg|webp)
        transformation:
          fetch_format: <format>

• transformation is one element of Cloudinary’s transformation array, or a list of them.
• Extensions are given in the path in the order they are declared. With the declaration above, eecd837c.200x200.jpeg matches and eecd837c.jpeg.200x200 does not.
• An entry marked optional: true may be left out of the path; any other must be present.
• A path with an extension that matches nothing, or missing a required one, is an error — 404 over HTTP.
• ^ and $ are added to the expression unless already there. zoom is an integer from 0 to 100. The format jpeg is sent to Cloudinary as jpg.

A fixed extension gives a named variant:

      - extension: icon
        transformation:
          width: 48
          height: 48
          crop: thumb
        optional: true

/products/kettle/eecd837c.icon.jpeg is then a 48-pixel thumbnail as JPEG.

Custom providers are not supported.

§A storage per environment

Like other Context keys, a storage can be declared differently per environment by suffixing its name. A common arrangement keeps files in a temporary directory locally and in a bucket in production:

# context.toa.yaml
storages:
  pictures:
    provider: tmp
    directory: store-pictures
  pictures@production:
    provider: spaces
    space: store-pictures
    region: fra1

Single options can be suffixed the same way: prefix@production: store/production. Environments are described in Environments.

§Secrets

A provider’s secrets are not written in the Context.

Locally they are environment variables named TOA_STORAGES_<STORAGE>_<SECRET>, in upper case, in the .env file that toa env maintains:

TOA_STORAGES_PICTURES_ACCESS_KEY_ID=…
TOA_STORAGES_PICTURES_SECRET_ACCESS_KEY=…

In a deployment they are a secret named storages-<storage>, deployed with toa conceal:

$ npx toa conceal storages-pictures ACCESS_KEY_ID=… SECRET_ACCESS_KEY=…

§Installing a provider’s SDK

The SDKs of cloud providers are not installed with the runtime. s3 and spaces need @aws-sdk/client-s3, @aws-sdk/lib-storage and @aws-sdk/s3-request-presigner; cloudinary needs cloudinary; fs and tmp need nothing.

A deployment installs what its storages need by itself: each composition the SDKs of the storages its components declare, and a process makes those storages and no other. A workspace installs the same set with toa npm. The generated package.json runs it after every npm install, as postinstall; run it yourself whenever a storage with a new provider is added:

$ npx toa npm -p .

-p is the path to the application. Without the packages a composition fails to start, naming the package and the version it wants.

§Try it

Give pictures the tmp provider, which needs nothing installed and no directory made:

# context.toa.yaml
storages:
  pictures:
    provider: tmp
    directory: store-pictures

With pictures and its describe as Declaring a storage has them, restart and call the operation from the command line once a file is there (Upload and Download Files puts one):

$ npm run restart
$ npx toa call pictures.describe "{ input: { path: '/pictures/eecd837c' } }"

Why files are handled apart from entity state is in Files.

§Summary