FlashAttention-3 hits 1.3 PFLOPS on H100, and speculative decoding can push LLM throughput up to 3.6x in production. Read now.
Why These Two Techniques Matter Together
Serving large language models at production scale is usually limited by how fast the system can generate each next token, not by how fast a single matrix multiply can peak on paper. Two complementary approaches address different parts of that bottleneck: FlashAttention-3 improves the attention kernel so each forward pass uses the GPU more efficiently, and speculative decoding changes the decoding schedule so the model can accept multiple tokens per expensive verification step when a cheap draft is right. Together they target both compute utilization and sequential latency—the two places inference spend tends to concentrate.
FlashAttention-3 is reported to reach 1.3 PFLOPS on H100 hardware, which signals that the attention path can stay closer to the chip’s practical peak instead of stalling on memory movement. Speculative decoding, in production settings, has been shown to raise LLM throughput by up to 3.6x when draft quality and batch shape align. Neither replaces the other: a faster attention kernel still helps every verified token, and speculation still needs a solid base model path underneath it.
FlashAttention-3: Faster Attention Without Changing the Math
Standard attention is memory-bound for long contexts: scores and intermediate tensors move between high-bandwidth memory and the compute units more often than necessary. FlashAttention-style kernels restructure the work into tiles that keep intermediate state in faster on-chip memory, fuse softmax and related ops, and reduce the volume of full attention matrices written back to HBM. FlashAttention-3 continues that line for current GPU generations, aiming for higher effective FLOPS on the same mathematical attention definition.
For practitioners, the practical takeaway is operational rather than theoretical. Drop-in kernels that preserve numerical intent let you keep existing model graphs while raising tokens-per-second on the same H100-class hardware. You still need to validate numerical parity on your stack, confirm that sequence lengths and head dimensions hit the optimized code paths, and measure end-to-end latency—not only microbenchmark FLOPS—because the rest of the pipeline (KV cache, sampling, networking) can hide kernel gains if left unoptimized.
Speculative Decoding: Spend Verification on Batches of Candidates
Autoregressive decoding is inherently sequential: each token depends on the previous one, so the large model’s forward pass runs once per step. Speculative decoding breaks that pattern by letting a cheaper draft model (or a draft head) propose several tokens ahead, then asking the large model to verify them in a single pass. Accepted tokens advance the sequence without extra large-model steps; rejected tokens fall back to normal sampling from that point.
Throughput gains scale with acceptance rate. When the draft stays close to the target model’s distribution—same domain, similar temperature, well-matched length of the draft window—you amortize one expensive forward over many tokens. When the draft drifts, rejections rise and you pay draft cost without much speedup. Production deployments that report up to 3.6x higher throughput typically combine a capable base model path with a draft strategy tuned for their traffic, not a one-size draft length applied blindly.
- Keep draft length short enough that acceptance stays high under your real prompts.
- Align draft and target sampling settings so rejected prefixes do not dominate.
- Measure tokens per second and tail latency under concurrent load, not only single-stream demos.
- Budget draft-model capacity separately; free draft cycles are not free if they steal GPU from verification.
Putting Both to Work in a Serving Stack
A sensible rollout treats FlashAttention-3 as a foundation and speculation as a scheduling layer. First, ensure the base model’s attention and KV-cache path use an efficient kernel on H100 so every verified token is as cheap as the hardware allows. Then introduce speculative decoding with offline acceptance curves on production-like prompts, and only then raise draft length or concurrency. Watch for interactions: longer contexts increase attention cost, which makes each verification more expensive and can change the break-even acceptance rate for speculation.
Instrumentation should track acceptance rate, mean tokens accepted per verify step, GPU utilization during draft versus verify, and end-to-end latency percentiles. If acceptance is high but utilization is low, you may have headroom to batch more requests or lengthen drafts carefully. If FLOPS look strong after FlashAttention-3 but throughput plateaus, the sequential decode schedule—not the kernel—is still the limiter, which is exactly where speculative decoding earns its keep. Used together, they turn peak attention performance and multi-token verification into a coherent path from kernel efficiency to production throughput.