Cold starts can differ by tens of milliseconds even on the same host. Benchmark Wasm vs Docker with a reproducible local harness today. Full breakdown.
Why cold-start latency needs a controlled comparison
WebAssembly runtimes and container engines both aim to ship isolated workloads, but they start from different models. Containers typically pull in a full process tree, filesystem layers, and network setup. Wasm modules usually load a compact binary into a sandbox with tighter system-call surface. That difference shows up most clearly at cold start—the time from “invoke this workload” to “it is ready to handle work”—and those gaps can be tens of milliseconds even on the same host when you control for CPU, memory pressure, and background noise.
A useful benchmark does not crown a permanent winner. It answers a narrower question: for a given workload shape and host setup, how long does each path take to become ready, and how stable is that number across runs? Latency tails matter as much as the median, because one slow cold start can dominate user-facing request time in short-lived or bursty services.
Build a reproducible local harness
Run both paths on one machine, same kernel, same power profile, and as few concurrent jobs as possible. Pin frequency scaling if your platform allows it, and warm the disk cache only when that matches your real cold-start story—otherwise force a true cold path so page cache does not silently favor the second run. Record wall-clock time from the client that issues the start, not from inside the guest, so you include runtime overhead that users actually pay.
Define the same application contract for both sides: identical request handling, identical response body size, and the same “ready” signal (listening port, health endpoint, or first successful request). Keep images and modules small and purpose-built for the test so packaging bloat does not masquerade as runtime cost. Script the sequence so you can re-run it without hand-clicking tools.
- Cold start: process or module not resident; measure invoke → ready.
- Warm start: already loaded; measure only request handling if you care about steady state separately.
- Many iterations: enough samples to see median, p95, and outliers, not a single lucky run.
- Fixed inputs: same payload, same concurrency (usually one request for pure cold-start), same timeout rules.
What to measure—and how to read the results
Capture at least three series: pure cold start to ready, cold start plus first request, and warm request latency. Log raw samples, then summarize with median and a high percentile. If Docker and Wasm both land in a similar band on your box, packaging, image pull policy, or host load may be dominating; if one path consistently wins cold starts but loses on warm throughput, that tradeoff is often the real decision input for serverless-style vs long-running services.
Watch for confounders: first-run module compilation, image layer extraction, antivirus scanners, and shared CPU steal on busy hosts. Discard or separately label the first sample if your pipeline always pays a one-time compile cost that production amortizes differently. Document host specs, runtime flags, and how you defined “ready” so the harness stays comparable when you change either stack later.
Turning numbers into a shipping choice
Use the harness to match deployment shape, not to chase a headline. Prefer the path whose cold-start distribution fits your traffic: frequent scale-to-zero favors the lower and tighter cold curve; always-on workers care more about warm latency and operational tooling you already trust. Re-run the same scripts after OS, runtime, or app changes—startup costs drift, and a local harness is cheap insurance against silent regressions.
Keep the benchmark local and scripted so any engineer can reproduce it today without cloud accounts or vendor dashboards. When results disagree with intuition, inspect the timeline (load, init, bind, first handler) before changing architecture. The goal is a clear, repeatable comparison of Wasm and Docker startup latency under conditions you control—not a universal ranking.