Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM

Nvidia published a deployment blueprint demonstrating how to serve the 753-billion-parameter GLM-5.3 NVFP4 mixture-of-experts model locally across two DGX Station systems using vLLM, as detailed in developer.nvidia.com's report. The quantized checkpoint requires 464 GB of GPU memory, surpassing the 252 GB HBM3e capacity of a single GB300 GPU. By linking two GB300 DGX Stations over a ConnectX-8 RDMA direct rail, engineers can host the complete model in high-bandwidth memory without CPU weight offloading, splitting tensor computation across both machines under tensor parallelism rank two.

This technical breakdown explains how to prepare dual DGX Stations, configure networking interfaces, synchronize model checkpoints, and benchmark the resulting OpenAI-compatible inference endpoint under concurrent load. The setup is designed for machine learning engineers, infrastructure architects, and enterprise developers who require frontier-scale reasoning throughput in local or air-gapped environments without external API calls.

01Dual-station memory partitioning and network rails

Running GLM-5.3 locally requires managing a 753-billion-parameter architecture that activates 40 billion parameters per token. Even with sparse expert activation routing tokens dynamically, all weights must reside in GPU memory to prevent execution stalls. Distributing the 464 GB model across two DGX Stations provides sufficient HBM3e capacity while dividing computational tasks evenly between the two GB300 accelerators.

The two systems connect over direct ConnectX-8 RDMA network rails identified as cx8r0 and cx8r1. Benchmarking engineers select one rail, such as cx8r1 mapped to the mlx5_1 device, providing a 400000Mb/s network link with an MTU setting of 9000. Jumbo-frame ping testing and NTP time synchronization ensure low-latency communication between Station A as rank 0 and Station B as rank 1 before tensor operations begin.

Station A operates the primary OpenAI-compatible API endpoint on port 8000, while Station B functions purely as a backend worker compute node. A shared environment configuration file defines node ranks, static IP addresses, RDMA device identifiers, and container images to ensure deterministic cluster initialization across both machines.

02Throughput performance across concurrent load tiers

Testing on the SPEED-Bench Coding dataset using AIPerf demonstrates how the dual-station deployment responds as client request concurrency scales. In inference benchmarks, the distributed vLLM setup achieved a peak token generation speed of 607.8 tokens per second at concurrency 16. Throughput dropped to 385.0 tokens per second when request concurrency was increased to 32, accompanied by a mean request latency of 453.5 seconds.

Memory consumption remains stable on both GPUs during sustained inference runs because CPU weight offload is explicitly disabled. The benchmark profile indicates that concurrency 16 delivers optimal performance for batch coding tasks, while higher concurrent queues degrade response latency as tensor parallel synchronization overhead increases.

03Container deployment and checkpoint synchronization

Setting up the serving stack begins by pulling identical container images on both nodes to avoid NCCL execution mismatches. Both Stations pull the official pinned aarch64 image digest from the vLLM container repository.

docker pull vllm/vllm-openai:v0.30.0-aarch64@sha256:4864d46625cbc3307623e29ac742030655e27249feba7b97ec925ce4cc4dfb56

Engineers then download the quantized checkpoint directly from Hugging Face on Station A to the local models directory.

docker run --rm --network host -v "$HOME/models/GLM-5.3-NVFP4:/model" \
 --entrypoint bash vllm/vllm-openai:v0.30.0-aarch64 \
 -lc "exec hf download 'nvidia/GLM-5.3-NVFP4' --revision 'e3b8f9f2fad29727dfa95e766c5cac18ea1414bf' --local-dir /model"

Once downloaded, the files are synchronized to Station B over SSH and rsync. The endpoint is launched with tensor parallelism set to two across the nodes, exposing the model identifier for inference clients.

curl http://192.168.101.1:8000/v1/models

Client applications can target the exposed service using standard OpenAI-compatible API parameters.

curl http://192.168.101.1:8000/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{"model": "nvidia/GLM-5.3-NVFP4", "messages": [{"role": "user", "content": "Write a test suite in Python."}]}'

04Interface verification and RDMA communication gotchas

A common failure mode in multi-station inference is NCCL timeout errors caused by misconfigured RDMA interfaces or MTU mismatches. Administrators must verify that the ConnectX-8 network interface link is active at 400000Mb/s and that jumbo frames pass between nodes using size 8972 payloads before starting containers.

Clock drift between the two DGX Stations can also disrupt distributed communication handshakes. Running timedatectl checks confirms NTP synchronization across both machines. Additionally, permissions for the Docker daemon must be configured correctly for non-root users by adding the user account to the local docker group.

Storage capacity represents another operational limit. Each machine must maintain at least 464 GB of free local storage space exclusively for its copy of the GLM-5.3 weights. Attempting to share weights over a networked file mount or enabling CPU offloading introduces severe I/O bottlenecks across tensor parallel rank barriers.

05Workload tuning and downstream deployment steps

Following cluster verification, developers must calibrate serving configurations against their target workload characteristics. Running AIPerf against custom prompt distributions helps identify whether token generation matches the sweet spot observed at concurrency 16 or requires queue throttling.

Enabling FP8 KV-cache storage, prefix caching, and multi-token prediction inside vLLM allows teams to maximize context processing efficiency under production traffic. These features reduce per-request memory pressure while preserving KV-cache state across repetitive prompt templates.

Teams planning enterprise rollouts should review the Connect Two DGX Stations validation playbook and consult the local AI recipe repository for validated network topologies. Verifying RDMA stability on secondary rails provides redundancy if primary network paths fail during continuous inference runs.