GitHub Copilot benchmarks site adds harness comparisons
GitHub launched a public preview update to its Copilot benchmarks site, adding Terminal-Bench-Science 0.1 and paired harness comparisons against Claude Code.
By Dillip Chowdary • Nov 18, 2026 • Source: GitHub Changelog

At a glance
- Product
- GitHub Copilot benchmarks site
- Update
- Harness comparisons view
- New benchmark
- Terminal-Bench-Science 0.1
- Harnesses compared
- Copilot CLI, Claude Code, Codex CLI
- Reasoning level
- Medium reasoning effort
- Status
- Public preview
In this post
GitHub launched a public preview update to its GitHub Copilot benchmarks site, introducing a dedicated harness comparison view and scientific research evaluation through Terminal-Bench-Science 0.1. According to GitHub Changelog's report, the refreshed benchmarking platform allows developers to compare how distinct agent runtimes execute tasks using identical underlying models. The tool tracks performance metrics across software engineering problems, weighing task resolution rates, reasoning effort levels, and mean financial cost per task.
This update directly serves engineering leads, tool builders, and developers evaluating agent harnesses such as GitHub Copilot CLI, Claude Code, and Codex CLI. By isolating harness scaffolding from base model capabilities, the site reveals how runtime orchestration influences accuracy, variance, and operational spend. Readers can inspect head-to-head metrics to determine whether scaffold optimizations deliver measurable cost reductions and task completion gains across standard software workloads and scientific domains.
01Public preview harness comparison view
GitHub expanded its Copilot benchmarking site to evaluate agent harnesses alongside standard foundation models. The interface evaluates GitHub Copilot CLI directly against Claude Code and Codex CLI under standardized conditions. Every comparison controls for reasoning effort, isolating how the surrounding client implementation alters practical output.
The platform visualizes results using a dual-axis scatter plot and a paired differences analysis. The scatter plot maps resolution rates against mean cost per task for each harness configuration. Shaded ellipses highlight run-to-run variation observed across published runs, while connecting lines link matched harness evaluations to make divergence immediately visible.
The paired differences view calculates GitHub Copilot CLI's resolution advantage in percentage points alongside relative cost savings expressed as a percentage. Both metrics display approximate 95% bootstrap intervals to indicate statistical significance across repeated evaluations.
02Terminal-Bench-Science research evaluation scope
Coverage on the platform now reaches beyond traditional software engineering repositories into complex scientific workflows. The site added Terminal-Bench-Science 0.1, an evaluation suite designed to test AI agents against multi-step research problems across technical disciplines.
The benchmark evaluates agent capabilities across life sciences, physical sciences, earth systems, mathematics, and engineering domains. Tasks are authored and reviewed by domain experts within each field to ensure challenges reflect authentic computational research rather than synthetic programming exercises.
Developers can inspect these domain-specific outcomes by selecting Terminal-Bench-Science 0.1 from the site benchmark picker. The interface reports whether agents sustain coherent terminal reasoning across specialized data processing pipelines and research environments.
03Controlled model matching and exclusion parameters
The comparison architecture enforces strict pairing criteria to avoid mismatched model distortions. GitHub limits side-by-side harness analysis to identical model configurations operating at medium reasoning effort.
Unmatched runs and proprietary configurations without exact peer counterparts are excluded entirely from the harness view. For example, GitHub Copilot Fusion and unique harness permutations are omitted rather than benchmarked against dissimilar base models.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
| Metric or condition | Copilot CLI paired view | Alternative harnesses |
|---|---|---|
| Reasoning effort setting | Medium | Medium |
| Model matching requirement | Exact pairs only | Exact pairs only |
| Evaluated tools | GitHub Copilot CLI | Claude Code, Codex CLI |
| Statistical error bounds | 95% bootstrap intervals | 95% bootstrap intervals |
| Unmatched configurations | Excluded (e.g. Fusion) | Excluded |
Note: Copilot-only configurations such as GitHub Copilot Fusion are excluded from paired harness comparisons to ensure every data point reflects identical base models.
04Inspecting published run data online
Developers can explore the complete dataset directly on the GitHub Copilot benchmarks site without installing local benchmarking tools. The web dashboard provides interactive filters to toggle between standard programming suites and scientific evaluation sets.
Users can isolate specific harnesses or examine multi-run variance across the published evaluation matrix. The interface groups runs so engineers can examine how client harnesses alter resolution accuracy on identical benchmarks.
- Access dashboard Navigate to the public GitHub Copilot benchmarks portal in a web browser.
- Select benchmark Choose Terminal-Bench-Science 0.1 or an established software suite from the benchmark picker.
- Open harness tab Switch to the Harnesses view to inspect paired scatter plots and bootstrap confidence intervals.
05Accessing benchmarked agent runtimes
The harness comparisons focus on command-line developer agents that run in terminal environments. Teams looking to test these environments locally can access GitHub Copilot CLI, Claude Code, or Codex CLI through their respective client update and launch commands.
To update or install Claude Code CLI for local agent workflows:
npm install -g @anthropic-ai/claude-codeTo switch models interactively or specify model selection within the agent interface:
/model claude-fable-5-1To launch a terminal session pinned to a specific model version directly from the command prompt:
claude --model fableFrequently asked questions
What does the new Harnesses view on the GitHub Copilot benchmarks site compare?
It compares GitHub Copilot CLI against Claude Code and Codex CLI using identical base models at medium reasoning effort, measuring resolution rate, reasoning effort, and mean cost per task.
What is Terminal-Bench-Science 0.1?
It is an evaluation suite added to the site that tests AI agents on scientific research workflows across the life, physical, earth, mathematical, and engineering sciences.
Why are configurations like GitHub Copilot Fusion omitted from the paired harness view?
The view only displays exact model pairs at medium reasoning effort, excluding unmatched configurations or Copilot-only setups to prevent misleading comparisons against dissimilar models.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement