Latency and Autoscaling Benchmark

SlimFaas includes a reproducible native-local benchmark for the warm synchronous proxy path, asynchronous enqueue and delivery, and scale-to-zero followed by PromQL scale-out.

Run the benchmark

The quick profile is intended for a smoke test and takes a few minutes:

BENCHMARK_PROFILE=quick .bin/slimfaas-local-benchmark.sh

Run the three-node recovery scenario

The restart-under-load scenario starts the same three-node SlimFaas Local cluster, writes continuously to SlimData, generates application HTTP traffic, kills a non-leader node, and fails unless the leader remains stable while the restarted node reports both /health and /ready successfully and committed Raft entries continue to advance:

RECOVERY_TIMEOUT_SECONDS=180 .bin/slimfaas-local-recovery-benchmark.sh

Use RECOVERY_SKIP_BUILD=true for repeated runs with existing Release binaries and RECOVERY_RUN_ROOT to select the artifact directory. The scenario requires lsof and writes recovery-summary.txt, traffic logs, and the local supervisor log below the selected artifact directory.

The standard profile runs every case three times with longer measurement and warm-up windows:

.bin/slimfaas-local-benchmark.sh

The async-queue profile uses the same payload/concurrency matrix and adds a low-rate paced scenario plus a burst of 1,000 messages at concurrency 64:

BENCHMARK_PROFILE=async-queue \
BENCHMARK_RUN_ROOT=artifacts/slimfaas-local-benchmark/async-baseline \
.bin/slimfaas-local-benchmark.sh

The script builds SlimFaas without the dashboard, builds the benchmark driver, validates benchmarks/slimfaas.local.benchmark.yaml, starts a clean three-node cluster with slimfaas local, runs the matrix, and shuts down every managed process. Docker and Kubernetes are not required.

It requires the repository's .NET 10 SDK plus Bash and curl. Ports 31020-31023, 3162-3164, 31080, and the 32000-32015 process pool must be free.

Results are written below artifacts/slimfaas-local-benchmark/<profile>-<UTC timestamp>/:

• summary.md: tables for sync overhead, async latency, and scaling milestones;
• results.json: structured settings and results;
• latency-samples.csv: one row per measured HTTP request;
• scaling-timeline.csv: observed replicas and queue depths over time;
• benchmark-manifest.txt: commit, dirty-worktree flag, SDK, host, and matrix;
• slimfaas-local.log: control-plane and managed-process output.

Compare an optimization with its baseline

Run the standard profile before changing the runtime, then run the exact same matrix after the change. Compare the two structured reports with:

dotnet run --project src/SlimFaasBenchmark -- compare \
  --baseline artifacts/slimfaas-local-benchmark/baseline/results.json \
  --candidate artifacts/slimfaas-local-benchmark/candidate/results.json \
  --output artifacts/slimfaas-local-benchmark/comparison

For an async queue optimization, capture the candidate on the same machine and compare with the async acceptance rules:

BENCHMARK_PROFILE=async-queue \
BENCHMARK_RUN_ROOT=artifacts/slimfaas-local-benchmark/async-candidate \
.bin/slimfaas-local-benchmark.sh

dotnet run --project src/SlimFaasBenchmark -- compare \
  --profile async \
  --baseline artifacts/slimfaas-local-benchmark/async-baseline/results.json \
  --candidate artifacts/slimfaas-local-benchmark/async-candidate/results.json \
  --output artifacts/slimfaas-local-benchmark/async-comparison

The command rejects reports whose duration, warm-up, repetitions, payloads, concurrency, or scaling settings differ. It writes:

• comparison.md: human-readable before/after latency, overhead, throughput, async, scaling, and acceptance verdicts;
• comparison.json: the same comparison as structured data.

The command exits with a non-zero status when the candidate records errors or timeouts, when the median added p50 reduction is below 20% for 64 B–4 KiB or 40% for 256 KiB–2 MiB, or when added p95/p99 or the throughput ratio regresses by more than 10%. Async latency and scaling timings are reported for context; only their errors and timeouts affect the verdict.

With --profile async, the verdict additionally requires no missing, duplicated, failed, or expired message; median reductions of at least 50% for the HTTP-202 p95 and 40% for the function-arrival p95; and at least 40% p50 improvement for 2 MiB. Each case is also guarded against a p99 or throughput regression above 10%. CPU per message may grow by at most 20%, peak memory by 10%, and Raft entries per message may not exceed 2x the baseline. Missing resource fields in an older results.json remain readable and are displayed as unavailable.

The synchronous and scaling sections are still generated by an async comparison to expose collateral changes. Sync microbenchmark guardrails are informational for --profile async; the async p99/throughput/resource rules and all correctness failures determine that profile's verdict. Use the default --profile sync comparison when sync guardrails must be gating.

What is measured

Warm synchronous overhead

The benchmark target is one minimal HTTP/1.1 server. The driver calls that exact process in two ways:

1. directly at http://127.0.0.1:31080/echo;
2. through /function/benchmark-latency/echo.

For each payload size and concurrency, the report includes throughput and HTTP p50, p95, p99, and maximum latency. It then subtracts the direct percentile from the SlimFaas percentile in milliseconds and percent. Calling the same process on both paths isolates the warm routing/proxy overhead from application work.

The standard matrix uses:

• payloads: 64 B, 4 KiB, 256 KiB, and 2 MiB;
• closed-loop concurrency: 1 and 16;
• warm-up: 2 seconds;
• measurement: 10 seconds;
• repetitions: 3.

The report uses the median percentile across repetitions and the mean throughput. The driver alternates direct/proxy order between cases to reduce systematic CPU frequency and temperature bias.

Asynchronous latency

Each async case reports two different clocks:

• acceptance latency: client send until SlimFaas returns HTTP 202 after the request has been durably enqueued;
• arrival latency: client send until the target handler starts, including enqueue, queue wait, dispatch, and request-body transfer.

For both clocks, the report gives the mean (the simple average) and p50/p95/p99. For example, p95 means that 95% of messages were no slower than that value. The mean is the easiest headline number, while p95 and p99 reveal occasional slow messages that an average can hide. The target also records completion latency in results.json. The target and driver are local processes on the same clock, so UTC ticks can be compared without a network clock-synchronization error. The benchmark waits for every accepted message to be observed by the target and for queue metrics to remain at zero before starting another case.

The 2 MiB case is intentionally above SlimFaas' 1 MiB async body-offload threshold. This makes the matrix cover both inline queue payloads and the cluster-file path.

The async profile also records throughput, missing and duplicate IDs, errors, expirations, Raft entries per accepted message, process CPU per message, and peak aggregate SlimFaas working set. The paced scenario measures how quickly a nearly idle queue wakes without waiting for the polling fallback. The burst scenario measures coalescing and backpressure under sudden load.

Scaling speed

benchmark-scale starts at zero replicas. The driver sends a burst of 200 async messages at concurrency 32; each target request takes 250 ms and each replica is limited to one concurrent request. The following durations are measured from the first client send:

• first and last HTTP 202;
• desired replicas changing from 0 to at least 1;
• first ready replica;
• desired replicas reaching 4;
• all 4 replicas becoming ready;
• the ready, in-flight, and retry queues returning to zero.

The first delivered /work request makes the benchmark target expose slimfaas_benchmark_scale_pressure 4 for 30 seconds. SlimFaas scrapes it every second and evaluates max_over_time(slimfaas_benchmark_scale_pressure[30s]). This deterministic demand signal exercises metric discovery, scraping, the internal metric store, PromQL evaluation, scale policy application, local process launch, and health readiness. Queue metrics are observed independently to measure backlog and drain time.

Configure the matrix

The wrapper accepts environment variables without changing tracked files:

BENCHMARK_PROFILE=quick \
BENCHMARK_DURATION_SECONDS=5 \
BENCHMARK_WARMUP_SECONDS=1 \
BENCHMARK_REPETITIONS=2 \
BENCHMARK_PAYLOAD_BYTES=64,4096,1048576,2097152 \
BENCHMARK_CONCURRENCY=1,8,32 \
BENCHMARK_SCALE_MESSAGES=500 \
BENCHMARK_SCALE_CONCURRENCY=64 \
.bin/slimfaas-local-benchmark.sh

Available variables are:

Interpret the result correctly

• Subtract percentiles in milliseconds first. A sub-millisecond direct baseline can make a small absolute delta look large as a percentage.
• Async HTTP 202 is enqueue latency, not execution latency. Use target delivery percentiles for end-to-end dispatch behavior.
• Native local scaling measures SlimFaas plus operating-system process startup. It deliberately excludes Kubernetes scheduling, admission, image pulls, CNI, and container runtime time. Run the same driver against a Kubernetes deployment when those costs are part of the question.
• Function status endpoints have short caches. Scaling times are externally observed milestones with that polling resolution, not internal timestamps.
• Avoid unrelated workloads, power-saving mode, thermal throttling, and debug builds. Keep the same host, SDK, commit, and matrix for before/after comparisons.
• A single run is exploratory. Use the standard repetitions, compare medians, and retain the raw manifest and samples for a publishable result.

See Native Local Development Mode for the process orchestrator and Autoscaling for the production scaling model.