Data temporary binary API

SlimFaas provides temporary binary storage through the /data/files endpoints. This feature is designed for internal workflows (functions, jobs, agents, pipelines) that need to store and retrieve binary artifacts (PDF, ZIP, audio, PPTX, etc.) for a limited time.

This page explains:

• how to use the /data/files API,
• how it works internally (cluster behavior, safety limits),
• how to configure Public vs Private access and override it via environment variables.

What is /data/files for?

Use /data/files when you need:

• a simple temporary store for binary artifacts
• a way to retrieve an artifact from any node in a SlimFaas cluster
• a solution that avoids shipping large binaries through the cluster’s consensus layer

Core idea

• The binary content is stored on disk.
• The metadata (checksum, size, content type, filename, TTL) is stored in a cluster-consistent store so every node knows “what exists”.

Follow a temporary artifact

An upload returns an ID you can pass to a function or job. Consumers use that ID through the API; metadata is replicated through Raft, while binary bytes travel directly between nodes when needed.

The file exercise uploads the supplied fixture and compares the downloaded bytes. The dashboard shows traffic; use API responses to verify the file itself.

API summary

Base path: /data/files

Store an artifact: POST /data/files

Request contract

• Body: raw bytes (not multipart/form-data)
• Content-Length is recommended (for accurate concurrency accounting)
• Recommended headers:
  • Content-Type (defaults to application/octet-stream)
  • Content-Disposition (to provide a download filename)

Query parameters:

• id (optional): artifact identifier
  • If omitted, SlimFaas generates an id.
  • Use URL-safe ids (letters, numbers, ., _, -).
• ttl (optional): time-to-live in milliseconds
  • Example: ttl=600000 (10 minutes)

Content-Length behavior (when missing or unknown)

SlimFaas streams the request body to disk. For its transfer safety guard (see 256 MiB parallel limiter), SlimFaas normally relies on Content-Length to “reserve” the right byte budget.

If Content-Length is missing or unknown (e.g., chunked transfer encoding), SlimFaas:

• accepts the upload
• logs a warning
• reserves the full default transfer budget (256 MiB) so an unknown-size upload runs alone

Recommendation Provide Content-Length whenever possible to get accurate concurrency behavior (especially behind proxies).

Responses

• 200 OK with the artifact id as plain text
• 400 Bad Request if the id is invalid

Examples (curl)

Store an artifact for 10 minutes and keep a filename:

curl -X POST "http://<slimfaas>/data/files?ttl=600000"   -H "Content-Type: application/pdf"   -H "Content-Disposition: attachment; filename=\"report.pdf\""   --data-binary @./report.pdf

Store an artifact with a chosen id:

curl -X POST "http://<slimfaas>/data/files?id=my-artifact-001&ttl=300000"   -H "Content-Type: application/octet-stream"   --data-binary @./payload.bin

Download an artifact: GET /data/files/{id}

What happens on a download

When you request an artifact:

1. SlimFaas applies a Raft read barrier and reads the cluster-consistent metadata for {id} (checksum, size, content type, filename, TTL)
2. It checks whether the binary exists locally and matches the expected checksum
3. If missing, SlimFaas attempts to pull the binary from another cluster node that has it
4. The response streams the binary to the client

After a create or overwrite returns 200 OK, a later download through any healthy cluster node uses the latest committed metadata. A follower waits until that metadata is applied locally instead of serving an older checksum.

Responses

• 200 OK streamed response
  • Content-Type from metadata (fallback application/octet-stream)
  • a download filename if provided at upload time
• 400 Bad Request if id is invalid
• 404 Not Found if metadata is missing/expired, or if no node can provide the binary
• 503 Service Unavailable if the Raft quorum cannot guarantee a current metadata read

Example (curl)

curl -L "http://<slimfaas>/data/files/my-artifact-001" -o my-artifact-001.bin

Delete an artifact: DELETE /data/files/{id}

This removes the cluster-consistent metadata and immediately attempts to remove the binary stored on the node handling the request. Other replicas remove their orphaned copy after three cleanup observations (at most about 90 seconds with the default interval).

• 204 No Content on success
• 400 Bad Request if id is invalid

Physical cleanup is best-effort: the metadata deletion remains authoritative and a background cleaner retries local orphan cleanup.

List artifacts: GET /data/files

Returns the list of known artifact ids (metadata-based) and their expiration information (if any). Expired entries are not returned.

Example:

curl -s "http://<slimfaas>/data/files"

Visibility & security (Public vs Private)

All /data/files routes are protected by a data visibility setting.

In appsettings.json:

{
  "Data": {
    "DefaultVisibility": "Private"
  }
}

Behavior

• Public: anyone who can reach SlimFaas can call /data/files
• Private (recommended): only internal requests are allowed External requests are typically answered with 404 Not Found (intentionally “hidden”).

Overriding visibility via environment variables

SlimFaas supports standard configuration overrides via environment variables.

To override:

• Data:DefaultVisibility

Use:

• Data__DefaultVisibility

Example: set /data/files to Public

Docker / Docker Compose

export Data__DefaultVisibility=Public

Kubernetes Deployment (excerpt)

env:
  - name: Data__DefaultVisibility
    value: "Public"

To keep it private (explicit):

env:
  - name: Data__DefaultVisibility
    value: "Private"

Notes

• This setting affects all data routes that are guarded by the same visibility filter, including /data/files.
• Use Public / Private for clarity.

How it works internally (high-level)

1) Metadata is cluster-consistent; content is disk-backed

For /data/files:

• metadata is stored in a cluster-consistent store so every node has the same view of what exists and how to serve it
• binary content is stored on disk on the node that received the POST request

This avoids moving large binaries through the cluster’s consensus layer. Metadata reads are linearizable: followers synchronize to the leader's current commit index before selecting the checksum to serve. Binary distribution remains eventual; an on-demand pull supplies the committed version when it is not yet present on the node handling the download.

2) Cluster distribution model: announce + pull

SlimFaas does not block the POST by pushing the binary to all nodes.

Instead:

1. The node that stored the artifact announces “I have {id} with checksum {sha}”
2. Other nodes can pull the artifact in the background
3. If a client requests an artifact from a node that doesn’t have it locally, that node can pull on demand

This yields eventual distribution with minimal upload latency.

Diagrams (simplified)

Step 1: Post /data/files

Step 2: Background announce handling

Step 3: Get /data/files/{id}

The 256 MiB parallel limiter (safety guard)

SlimFaas applies a safety guard that limits concurrent binary transfers to roughly:

• 256 MiB in parallel

Meaning:

• each in-flight transfer reserves a “byte budget” equal to its size
• new transfers wait if starting them would exceed the budget
• a single artifact larger than 256 MiB can run only if it is alone

When Content-Length is missing/unknown, SlimFaas reserves the complete default budget (256 MiB) and logs a warning.

This is not a strict memory cap; it’s a concurrency limiter.

Storage & durability

/data/files is disk-backed:

• If storage is ephemeral, artifacts may disappear on pod restart.
• For durability, mount a persistent volume.
• Artifacts can be re-pulled from other nodes only while at least one node still has the content and metadata still exists.

Troubleshooting

Upload logs a warning about Content-Length

Your client/proxy sent the request without Content-Length (often chunked transfer).

SlimFaas still accepts the upload and reserves the full 256 MiB default budget. The upload runs alone by default. Provide a known length (e.g. curl --data-binary @file) to allow safe parallel transfers.

File transfer memory settings

The defaults can be overridden with SlimData__Files__... environment variables:

{
  "SlimData": {
    "Files": {
      "MaxInFlightBytes": 268435456,
      "UnknownLengthReservationBytes": 268435456,
      "MaxPendingTransfers": 128,
      "QueueWaitTimeoutSeconds": 30,
      "DropPageCache": true
    }
  }
}

When the pending queue is full or a transfer waits longer than the configured timeout, SlimFaas returns 429 Too Many Requests with Retry-After: 1.

404 Not Found on download

• metadata expired or was deleted
• no node currently has the binary (restart, cleanup, ephemeral storage)
• temporary unavailability during cluster convergence

Quick reference

• Store: POST /data/files?id={id?}&ttl={ms?}
• Download: GET /data/files/{id}
• Delete: DELETE /data/files/{id}
• List: GET /data/files
• Visibility config: Data:DefaultVisibility = Public | Private
• Env override: Data__DefaultVisibility=Public

Live metadata inventory

The built-in dashboard's Live Stream → Data tab shows file keys, TTL and document sizes without fetching values or documents. Search by prefix, inspect expiration and navigate pages of 100 entries. The inventory updates at the configured status interval; short-lived entries between snapshots may be missed.

Metadata follows the existing data visibility policy unless SlimFaas__ExposeDataMetadata=true explicitly enables metadata access for dashboard visitors. This does not grant access to stored contents. See the dashboard inventory and SSE API.