AWS introduces Amazon SageMaker JumpStart Model Leaderboard
AWS launched the Amazon SageMaker JumpStart Model Leaderboard to compare 70+ foundation models on quality, speed, and cost directly in SageMaker Studio.
By Dillip Chowdary • Nov 19, 2026 • Source: AWS Machine Learning Blog

At a glance
- Product
- Amazon SageMaker JumpStart Model Leaderboard
- Availability
- Amazon SageMaker Studio
- Model catalog
- 70+ foundation models
- Providers
- Meta, Google, Qwen, IBM, Arcee
- Default profile
- Max Context with Balanced optimization
- Pricing basis
- On-demand hourly instance rate
In this post
Amazon Web Services launched the Amazon SageMaker JumpStart Model Leaderboard, introducing side-by-side evaluations of foundation models directly within Amazon SageMaker Studio. According to AWS Machine Learning Blog's report, the new tool ranks open-weight models across quality benchmarks, response latency, and operational expense without requiring users to answer preliminary setup questions. The feature eliminates the need to cross-reference disparate model cards by providing standardized evaluation data gathered under uniform testing conditions.
This report outlines the leaderboard's core metrics, configuration menus, underlying evaluation methodology, and step-by-step deployment workflow. Machine learning engineers, technical architects, and enterprise teams seeking to balance inference performance against infrastructure spend can use this breakdown to shortlist and launch models efficiently.
01Seventy foundation models in Studio
The new leaderboard provides a dedicated view on the JumpStart base models page in Amazon SageMaker Studio, positioned beside the existing Categories and All models tabs. It displays an immediate ranking of large language models while retaining familiar search bars and property filters. The interface allows team members to share direct links to specific comparisons through unique view URLs.
At launch, the catalog evaluates more than 70 foundation models from providers including Meta, Google, Qwen, IBM, and Arcee, with additional options scheduled over time. AWS measures each entry using two distinct data streams: standardized public quality benchmarks and operational measurements derived from actual endpoints. Combining these signals into a unified console removes the friction of assembling inconsistent claims from external websites.
Evaluation uniformity serves as the foundation for the comparisons. AWS executes each quality assessment using a consistent evaluation harness and identical generation settings across all listed models. Performance and hardware costs reflect measurements taken by transmitting a standardized test workload to endpoints created with standard JumpStart deployment configurations.
02Three standardized reasoning benchmarks
The system tests large language models against three public evaluation suites to establish comparable quality scores. Rather than blending distinct results into an aggregated score, the interface presents each benchmark as an independent column. This transparency allows engineering teams to inspect individual ratings against public specifications and prioritize tests aligned with their target production tasks.
Every benchmark targets complex reasoning abilities that challenge standard language models across varied cognitive tasks. Higher scores indicate superior capability across all three standardized evaluations. The tests measure multi-step arithmetic, scientific domain expertise, and broad academic knowledge to separate shallow pattern matching from substantive analytical proficiency.
| Benchmark | Target capabilities |
|---|---|
| BIG-Bench Hard (BBH) | Multi-step reasoning tasks including logical deduction, object tracking, and multi-step arithmetic |
| GPQA Diamond | Graduate-level science questions resistant to direct web searches, signaling deep reasoning |
| MMLU-Pro | Harder multiple-choice questions spanning broad knowledge and reasoning across science, law, and engineering |
03Workload profiles and optimization goals
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
Inference performance and operating expenses fluctuate depending on traffic characteristics and deployment settings. Two dropdown selectors positioned above the table allow users to adjust the reported operational data to reflect planned production scenarios. Changing these selections updates the latency and expenditure figures across the grid while leaving the underlying quality benchmark scores unchanged.
The use case profile menu defines the test workload, defaulting to the Max Context setting. Next to it, the optimization goal menu determines which specific endpoint configuration displays for a selected model. By switching optimization targets, teams can evaluate how specific architectural adjustments alter hardware requirements and token economics.
| Optimization goal | Deployment configuration |
|---|---|
| Balanced | Configuration with the lowest estimated cost per token, balancing hourly instance price against token throughput |
| Cost optimized | Configuration with the lowest hourly instance price, allowing higher latency and lower throughput |
| Latency optimized | Configuration with the lowest time to first token |
| Throughput optimized | Configuration with the highest throughput, which can use speculative decoding |
04Response latency and instance cost metrics
The leaderboard assesses deployment configurations against three core operational metrics gathered under standardized testing workloads. Time to first token measures how many milliseconds or seconds elapse before an endpoint initiates output, which serves as a critical indicator for interactive customer chat. Throughput per request records the tokens generated per second, representing the primary efficiency metric for high-volume document pipelines.
Instance cost calculations indicate estimated hourly expenses based on the specified hardware type tied to each deployment recipe. The configuration assigned to the Balanced goal provides the lowest cost per token, whereas latency-focused targets prioritize rapid responsiveness over raw instance pricing.
Note: Estimated cost reflects Amazon SageMaker instance expenses only, calculated from the on-demand hourly rate of the configuration's default instance type. This estimate excludes data transfer, persistent storage, inference component fees, and volume savings from Reserved Instances or AWS Savings Plans. Teams must verify projected billing totals through the AWS Pricing Calculator before committing production budgets.
05Five steps to deployed endpoints
Moving from model evaluation to a running SageMaker endpoint follows a structured five-step workflow in SageMaker Studio. Engineers begin with administrative prerequisites, requiring active console access alongside permissions to launch JumpStart resources. The integrated interface removes external tooling dependencies by executing discovery, analysis, and resource provisioning inside a unified management workspace.
- Open the leaderboard Navigate to the Models page within Amazon SageMaker Studio and choose the Leaderboard view to inspect ranked models or capture the direct page URL.
- Narrow the list with filters Apply licensing, provider, and modality constraints through the search bar to isolate candidates that satisfy organizational and legal requirements.
- Choose a use case profile and optimization goal Adjust the workload menu from Max Context and select an optimization objective such as Balanced, Latency optimized, or Throughput optimized.
- Read the heatmap and customize columns Scan blue column heatmaps where deeper shades signify superior metrics, reorder columns by sorting, and adjust visible metrics using the gear icon.
- Deploy or customize the model Open the selected model entry to execute serverless fine-tuning or deploy the endpoint using the identical profile and optimization configuration evaluated on the leaderboard.
Choosing the matching use case and optimization goal during endpoint creation ensures that production runtime characteristics align with leaderboard benchmarks. Deviating from the evaluated hardware configuration alters quality, latency, token throughput, and operational expenditure. For models that support serverless customization, teams can launch fine-tuning jobs directly from the model page before endpoint deployment.
Frequently asked questions
Which foundation models are included on the JumpStart Model Leaderboard?
The leaderboard covers more than 70 foundation models at launch from providers such as Meta, Google, Qwen, IBM, and Arcee, with additional models planned over time.
Which benchmarks does AWS use to score model quality?
AWS evaluates all models using three public reasoning benchmarks: BIG-Bench Hard for multi-step reasoning, GPQA Diamond for graduate-level science, and MMLU-Pro for broad academic knowledge.
How does the leaderboard calculate instance costs?
Estimated costs reflect on-demand hourly rates for the default Amazon SageMaker instance type of a selected deployment configuration. They exclude data transfer, storage, inference component fees, and savings plans.
What optimization goals can users select for model deployment?
Users can choose Balanced for lowest cost per token, Cost optimized for lowest hourly price, Latency optimized for lowest time to first token, or Throughput optimized for maximum generation speed.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement