Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6

The test: Benchmarking small LLM inference vs G5 and G6

AWS Machine Learning Blog reports: Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6. Benchmark two 30B Mixture-of-Experts models, Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B, across G5, G6, G6e, and G7 GPU instances on Amazon SageMaker AI. Compare throughput, latency, and cost-per-token, and see how G7's NVIDIA Blackwell GPUs deliver measurable price-performance gains for real-time LLM inference.

A single generation jump can slash latency, increase throughput, and reduce cost-per-token. However, the real-world magnitude of those gains depends on model architecture, quantization format, and workload shape.

How Benchmarking small LLM inference and G5 and G6 each did

In this post, we benchmark two representative 30B Mixture-of-Experts (MoE) models across three GPU instance families on Amazon SageMaker AI Inference using Amazon SageMaker AI Generative AI inference recommendation. This feature provides both recommendations for throughput, cost, and latency, and benchmarking for common metrics such as time to first token and latency.

Benchmarking small LLM inference vs G5 and G6, side by side

Using this feature, we demonstrate how the new G7 instances powered by NVIDIA Blackwell GPUs deliver measurable gains in throughput, latency, and cost-per-token. See the full write-up from AWS Machine Learning Blog via the source link for quotes and complete context.

Use case 1: AI coding assistant Deploy Qwen3-Coder-30B for enterprise coding tasks such as code generation, debugging, refactoring, and developer copilots. Benchmark ml.g5.12xlarge (A10G), ml.g6.12xlarge (L4), and ml.g7.12xlarge (RTX PRO 4500 Blackwell) instances using the Amazon SageMaker AI DJL Large Model Inference (LMI) container to compare price and performance.

See the Benchmarking small LLM inference vs G5 and G6 output

The G5 and G6 configurations each use four GPUs with 96 GB of aggregate GPU memory, while G7 uses two GPUs with 64 GB. This comparison shows how G7 performs with half the number of accelerators and less total GPU memory.

The verdict on Benchmarking small LLM inference vs G5 and G6

Use case 2: Enterprise AI assistant Deploy NVIDIA Nemotron-3-Nano-30B-A3B-NVFP4 for reasoning, question answering, summarization, and agentic workloads. See the full write-up from AWS Machine Learning Blog via the source link for quotes and complete context.