7B models can miss latency SLOs by 10x on cold boot without weight streaming and warm pools. See the architecture, metrics, and fixes. Read now.
Why Cold Starts Blow Past Latency SLOs
A 7B model is small enough to run on a single modern GPU, but large enough that loading weights, kernels, and runtime state is not free. In a serverless setup the first request after scale-to-zero must acquire capacity, pull the model into GPU memory, initialize the inference stack, and only then produce tokens. That path routinely dominates end-to-end latency. When the warm path meets a tight SLO, the cold path can miss it by roughly 10x—not because inference is slow, but because boot work was never part of the budget.
Treat cold start as a first-class latency path, not an ops edge case. If you only measure p50 on warm instances, you will ship a system that looks fine in dashboards and fails for the users who hit empty pools.
Architecture: What Actually Happens on Boot
Cold start is a pipeline, not a single event. Capacity must be scheduled; the container or sandbox must start; drivers and runtime libraries must load; model weights must move from object storage or a local cache into GPU memory; and the serving process must compile or load kernels before the first forward pass. Any stage that serializes the rest becomes the bottleneck. For 7B models, weight transfer and memory placement usually dominate once the runtime is already warm on the image.
Design the serving unit so those stages are explicit and measurable. Separate the immutable runtime image from the model artifact. Keep weights addressable as a versioned blob so you can stream or stage them independently of code deploys. Prefer a thin process that can start accepting connections early and report readiness only after GPU memory is fully resident and a smoke inference has completed.
Metrics That Matter
Instrument the cold path in stages rather than a single “startup time” number. Track time to capacity, time to process ready, time to weights resident, time to first successful inference, and time to first user-visible token. Record whether the request hit a cold instance, a warm idle instance, or a busy instance that had to queue. Without that split, you cannot tell whether the fix is faster packing, more headroom, or better weight loading.
Set SLOs on the cold path deliberately. If product latency assumes warm serving, define how often cold is allowed and what fallback applies when it is not. Alert on cold-start rate and cold p95, not only on average request latency.
Fixes: Weight Streaming and Warm Pools
Weight streaming reduces the long silent wait while a multi-gigabyte checkpoint downloads and then copies to the GPU. Stream shards in order of use, overlap network I/O with GPU placement, and pin the hot layers first so early tokens can start while later layers finish loading when your stack supports partial readiness. Cache weights on the node or in a fast local store so repeat cold starts on the same host skip the remote fetch. Avoid full-file materialization to disk when direct or staged GPU load is available.
Warm pools absorb the residual boot cost you cannot eliminate. Keep a minimum number of instances already loaded with the target 7B weights, sized to expected cold traffic rather than peak load. Scale the pool on leading indicators—queue depth, arrival rate, scheduled traffic—not only on CPU or GPU utilization after the fact. Combine a small always-warm base with burst scale for spikes so most user traffic never pays the full cold path, while you still retain serverless economics for idle periods.
- Stage weights close to the GPU and stream in dependency order instead of blocking on a full download.
- Hold a small warm pool of fully loaded 7B workers for SLO-critical traffic.
- Measure cold vs warm latency separately and gate releases on cold-path regressions.
Together, streaming shortens each cold boot and warm pools reduce how often users see one. That pairing is the practical fix when 7B serverless serving must stay inside a latency SLO that the naive cold path cannot meet.