Karpenter, KEDA, and vLLM let EKS add GPU nodes only when inference demand spikes, cutting idle burn while preserving throughput. Full breakdown.
Why idle GPUs dominate inference cost
GPU capacity for LLM inference is expensive when it sits unused. Fixed node pools sized for peak traffic keep cards warm for low latency, but most hours the cluster is far below that peak. The alternative is not “no GPUs,” but GPUs that appear only when requests justify them and leave when they do not. On EKS, that pattern is practical: treat the GPU fleet as a demand-driven resource rather than a permanent reservation.
Serverless here means operational, not magic. You still choose instance families, drivers, and model packing rules. What changes is the control loop: scale signals drive node creation, pods schedule onto those nodes, and when demand falls the nodes drain and terminate so you stop paying for idle accelerators.
vLLM on Kubernetes: the inference plane
vLLM serves as the model-serving layer that turns a GPU into an efficient inference endpoint. It batches concurrent requests, manages KV cache, and exposes an HTTP API that your app or gateway can call. In Kubernetes you run it as a Deployment or similar workload with GPU resource requests so the scheduler only places pods on nodes that advertise the right devices.
Pack models carefully: one large model per GPU is common, but multi-model or multi-replica layouts depend on memory headroom and latency targets. Health checks, readiness gates, and a sensible max concurrency limit matter more than clever YAML. If a pod is ready before weights are loaded, you get timeouts at the worst moment—right when a scale-up just finished.
KEDA for demand signals, Karpenter for GPU nodes
KEDA watches metrics that reflect real inference load—queue depth, HTTP concurrency, custom app metrics—and scales the vLLM replica count up or down. Replica count alone is not enough if the cluster has no GPU nodes. That is where Karpenter fits: when pending pods request GPU resources, Karpenter provisions the right node type, attaches it to the cluster, and later removes it when nothing needs it.
Wire the two so they reinforce each other rather than race:
- Define scale triggers that rise early enough for cold-start time (image pull, node bootstrap, model load).
- Request GPUs explicitly on the inference pods so Karpenter can select GPU-capable capacity instead of generic compute.
- Set scale-down and consolidation behavior so empty GPU nodes do not linger, without yanking capacity mid-request.
- Keep a small warm buffer only if your latency SLO cannot tolerate full cold starts; otherwise prefer true zero when idle.
Operational tradeoffs and a workable layout
Cold start is the main cost of elasticity. Node launch, driver readiness, and model load all add delay before the first token. Mitigate with pre-pulled images where possible, right-sized instance choices, and scale policies that react to leading indicators (queue growth) rather than only saturated CPUs. Throughput during spikes stays high because Karpenter can add several GPU nodes in parallel while KEDA adds replicas to match.
A clean layout separates concerns: a non-GPU control plane and app tier always on; a GPU inference Deployment scaled by KEDA; Karpenter provisioners or NodePools constrained to GPU instance types and the availability zones you actually need. Monitor pending GPU pods, scale events, and end-to-end latency together—if pods stay Pending while Karpenter is quiet, your resource requests or constraints are wrong; if nodes appear but latency still spikes, the bottleneck is model load or batching, not capacity. Used this way, Karpenter, KEDA, and vLLM give EKS a path to spike-ready LLM inference without permanent idle GPU burn.