Speculative decoding can cut per-token latency by 2-3x without changing output distribution, making sub-10ms edge LLMs feasible in production. Read now.
Why Per-Token Latency Dominates Edge Inference
On-device and near-edge LLM serving is rarely limited by how many parameters you can load. It is limited by how long each generated token takes to leave the model. Autoregressive decoding is sequential: every new token depends on the full prefix, so wall-clock time scales with sequence length even when the silicon still has spare FLOPs. For interactive assistants, keyboard prediction, and on-device agents, that sequential tax is what keeps responses feeling sluggish.
Sub-10ms per-token budgets are aggressive. They leave almost no room for wasted memory traffic, oversized drafts, or verification that serializes behind the draft step. Speculative decoding attacks that bottleneck directly: it reduces the number of expensive sequential steps without changing which tokens the large model would have produced.
How Speculative Decoding Works
The core idea is simple. A small, fast draft model proposes several tokens ahead. The large target model then checks those candidates in one parallel forward pass over the extended prefix. Accepted tokens are kept; the first rejected token is corrected by sampling from the target model, and drafting restarts from there. Because acceptance is decided by the target distribution, the final output distribution matches ordinary sampling from the large model alone.
That distribution-preserving property matters in production. You get lower latency without trading accuracy for speed, without inventing a new decoding policy, and without asking product teams to re-validate generation quality for every temperature or top-p setting. Throughput gains come from verifying multiple draft tokens per target-model call; quality stays tied to the model you already trust.
What Makes Sub-10ms Feasible on Edge Hardware
Edge devices win when draft work is cheap relative to target work and when acceptance rates stay high enough that most draft tokens survive. Practical setups pair a much smaller draft with a larger target that shares tokenizer and vocabulary so verification stays simple. Keep draft lengths short enough that a rejection does not throw away too much work, but long enough that you amortize the target-model pass across several tokens.
- Prefer a draft model that is fast on the same accelerator and memory hierarchy as the target.
- Cap speculative depth so failed drafts fail cheaply.
- Measure acceptance rate on your real prompts; chatty, structured, and code-like traffic often accept more than open-ended free text.
- Budget memory for both models resident, or for a shared backbone if you use a draft head on the target.
- Profile end-to-end per accepted token, not just model FLOPs—memory bandwidth and kernel launch overhead dominate at low latency.
A 2–3x cut in per-token latency is the range that turns “almost interactive” edge demos into production paths. If your baseline already sits in the low tens of milliseconds, that improvement is what puts sub-10ms within reach rather than requiring a wholesale model shrink that would hurt quality.
Production Tradeoffs and Failure Modes
Speculative decoding is not free. You pay for draft computation even when tokens are rejected. On very short replies, setup cost can erase the gain. On prompts where the draft and target disagree often, acceptance collapses and you fall back toward single-token latency plus draft overhead. Batching multiple users can also reduce headroom for speculative depth if the target pass is already memory-bound.
Treat it as a systems feature with clear knobs: draft size, max speculative tokens, fallback to greedy or standard sampling under load, and telemetry on acceptance rate, tokens per target step, and p50/p95 per-token latency. Start with the traffic that looks most predictable, keep the target model as the single source of truth for sampling, and only widen draft depth once acceptance and memory pressure look healthy. Done that way, speculative decoding is a concrete path to sub-10ms edge LLMs without rewriting how your product samples text.