Netflix In-House LLM Serving [Engineering]
Dillip Chowdary
July 20, 2026 · 7 min read · Source: Netflix Tech Blog
Bottom Line
Netflix’s in-house LLM serving post is a blueprint for when API-only inference is not enough: control latency SLOs, data gravity, and unit economics at streaming scale.
Key Takeaways
- ›Hosted APIs are fine until latency, data residency, or cost curves break product SLOs.
- ›Separate model runtime concerns from product features — a dedicated inference platform team pays off.
- ›Measure goodput (successful tokens under SLO), not just GPU utilization.
- ›Start with a narrow high-ROI workload before “platformizing” every LLM call.
In “In-House LLM Serving at Netflix” (July 17, 2026), Netflix’s AI Platform Model Runtime and Inference teams explain why they serve LLMs themselves rather than relying only on hosted APIs.
When self-hosting wins
- Latency SLOs for interactive product surfaces
- Data control for internal corpora and member-sensitive features
- Cost at volume once QPS is high and steady enough to fill accelerators
- Customization of batching, caching, and routing policy
Platform shape (practical takeaways)
Even without copying Netflix’s exact stack, the organizational split matters:
- Model Runtime — packaging, versions, rollout, canaries
- Inference — scheduling, batching, hardware efficiency, autoscaling
- Product teams — prompts, evals, UX — not CUDA tuning
Metrics that matter
| Metric | Why | Anti-pattern |
|---|---|---|
| p95 latency under SLO | User-visible quality | Only tracking mean latency |
| Cost / successful task | Business viability | Cost / GPU-hour alone |
| Cache hit rate | Prompt/prefix reuse savings | Ignoring repeat traffic |
| Rollback time | Safe model deploys | Big-bang model swaps |
Adoption path for smaller teams
- Pick one latency-sensitive or high-QPS workload
- Benchmark hosted API vs. self-host on that workload only
- If self-host wins, invest in runtime + eval harness before adding models
- Keep a hosted fallback for burst and disaster recovery
Read the full Netflix post for their internal context; mirror the principles, not necessarily the hardware list.
Primary source:
Netflix Tech Blog →
Verify claims against the original report before changing production systems.