Why goodput matters more than throughput for LLM serving

The **CNCF Blog (Eng)** published an article titled **Why goodput matters more than throughput for LLM serving**. The benchmark analysis notes that when evaluating an **LLM serving** setup, the primary metric almost everyone reaches for first is **throughput**, specifically measuring how many **requests per second** the system can push through.

Mechanically, **throughput** serves as the standard benchmark metric because it quantifies the volume of **requests per second** moving through an **LLM serving** system. As detailed in the source, this metric is easy to measure and easy to compare across setups. However, raw throughput metrics do not distinguish between valid serving performance and degraded outputs, making **goodput** the more critical metric for evaluating useful system work.

For engineers and system builders, relying exclusively on **throughput** creates a blind spot during performance testing. While achieving high **requests per second** appears strong because it is easy to measure and compare, builders evaluating an **LLM serving** setup must focus on **goodput** to ensure the serving infrastructure fulfills actual workload requirements efficiently.

In the broader market context of **LLM serving** infrastructure, benchmarks have historically emphasized raw **throughput** due to its simplicity. The **CNCF Blog (Eng)** position highlights a shift where evaluating **goodput** offers a more accurate comparative picture between competing serving setups than standard **requests per second** figures.

The practical takeaway for technical teams is to adjust benchmark criteria when assessing an **LLM serving** configuration. Rather than selecting **throughput** as the sole success metric because it is easy to compare, teams must measure **goodput** to validate real serving efficiency.

What to watch next is how benchmarking tooling and frameworks for **LLM serving** adopt **goodput** as a primary metric alongside raw **throughput**. Observing whether testing suites incorporate metrics beyond **requests per second** will indicate how effectively serving evaluations reflect true system performance.