TTFT benchmark: LLM Gateway vs. OpenRouter (Claude-haiku-4.5, 150 runs)

I'll pull the gist for the actual TTFT numbers so the paragraphs stay factual and specific.A GitHub gist by smakosh published a **TTFT** head-to-head of **LLM Gateway** versus **OpenRouter** on **claude-haiku-4.5**, using 75 cold and 75 warm runs per gateway (300 requests total) with zero errors. The run was taken on 2026-07-22 from a single residential connection. End-to-end median cold **TTFT** was 906ms on LLM Gateway and 1392ms on OpenRouter; warm medians were 814ms and 1232ms. That is about 35% faster cold and about 34% faster warm for LLM Gateway at the median. The write-up was also linked on Hacker News (item 49009590) with 1 point and 0 comments at the time of the listing.

The harness is rbadillap/ai-gateways-benchmark: Python stdlib only, raw sockets, with each request split into **DNS**, **TCP**, **TLS**, **TTFB**, and **TTFT** so connection setup is not mixed with gateway or model latency. Both sides used the same short prompt ("Reply with the single word: pong"), **max_tokens=16**, and interleaved round-robin cold/warm pairs so time-of-day drift hit both equally. Cold means a fresh connection (full DNS+TCP+TLS, new TLS context, no session resumption); warm is a second request on an already-open socket. Hosts were api.llmgateway.io (/v1/chat/completions, model anthropic/claude-haiku-4-5) and openrouter.ai (/api/v1/chat/completions, model anthropic/claude-haiku-4.5).

Phase medians show where the gap sits. LLM Gateway: DNS 3.2ms, TCP 20.9ms, TLS 40.2ms, TTFB 200.9ms, TTFT 829.5ms, cold e2e TTFT 906.1ms, warm TTFB 175.7ms, warm TTFT 813.8ms. OpenRouter: DNS 3.4ms, TCP 7.2ms, TLS 12.7ms, TTFB 1369.4ms, TTFT 1369.8ms, cold e2e TTFT 1392.0ms, warm TTFB 1228.7ms, warm TTFT 1232.1ms. Spread (median, p10–p90): cold TTFB 201 (194–240) vs 1369 (979–1643); cold e2e TTFT 906 (803–1380) vs 1392 (1002–1675); warm TTFB 176 (171–193) vs 1229 (924–1500); warm TTFT 814 (673–1379) vs 1232 (924–1501). LLM Gateway’s p90 cold e2e TTFT (1380ms) still sits below OpenRouter’s median.

For engineers building chat UIs, agents, or tools that stream tokens, first-token latency is what the user feels before any content appears. A ~500ms cold and ~400ms warm median gap on a small, fixed prompt is large enough to matter for short-lived workers and serverless paths that cannot keep a warm pool. The TTFB pattern is the practical architecture lesson: LLM Gateway starts the response stream around 200ms, before the first upstream token, with warm TTFB tightly banded (171–193ms p10–p90). OpenRouter’s first byte arrives with the first token (TTFB ≈ TTFT on every run), so it effectively holds the HTTP response until the upstream answers. TTFT remains the fair product number; TTFB mainly reveals who emits headers early.

Market context is not a pure same-backend race. The gist notes different default upstreams for this model: a sample OpenRouter request was served via Amazon Bedrock, while LLM Gateway runs were pinned to Anthropic’s API. OpenRouter’s edge for these runs terminated in Copenhagen (cf-ray …-CPH). OpenRouter is the better-known multi-provider router; LLM Gateway is the faster path in this single-vantage, default-routing setup. Neither result is a global ranking—the benchmark README itself frames results as a property of where you measure from.

Reproduce with the public repo and a config of 75 cold / 75 warm runs, the same one-word prompt, max_tokens=16, and both API keys. What to watch next: re-run from other regions and networks; pin both gateways to the same Anthropic upstream and re-measure; compare cold vs warm under real connection-pool lifetimes; and treat early-header streaming vs hold-until-token as a product choice when you pick a gateway for interactive work.